Article management method, device and smart refrigerator

Abstract

The present application provides an item management method, device and smart refrigerator, the method comprising: collecting an image sequence when a user manages items in an item storage device, wherein the image sequence includes a user's hand image area; tracking the hand image area to obtain the user's management behavior on the item and whether the user performs a code scanning operation on the item; determining the actual type of the item according to whether the user performs a code scanning operation on the item; pushing prompt information and updating the current stored item set of the item storage device, wherein the prompt information is used to indicate the actual type of the item and the user's management behavior. This greatly simplifies the operation process when the user manages items, thereby greatly improving the user experience.

Description

Technical Field

[0001] The present application relates to smart home appliance technology, and more particularly to an item management method, device and smart refrigerator. Background Art

[0002] With the continuous development of smart appliance technology, smart refrigerators have emerged. Compared with traditional refrigerators, smart refrigerators use cameras, barcode scanning modules, voice acquisition modules, display screens and other components installed on the refrigerator to help users manage food materials such as food storage and food removal, as well as provide users with food processing method inquiries.

[0003] In the prior art, a barcode scanning module can be set on a smart refrigerator, and the smart refrigerator manages the ingredients by scanning the barcodes of the ingredients based on the barcode scanning module on the refrigerator. Specifically, before the user stores ingredients in the smart refrigerator, he first triggers the barcode scanning module of the smart refrigerator to start working through voice or interface operation, and the barcode scanning module scans the information of the stored ingredients and completes the process of storing the ingredients. Correspondingly, before the user takes out an ingredient from the smart refrigerator, he first triggers the barcode scanning module to start working through voice or interface operation, and the barcode scanning module scans the information of the taken out ingredients and completes the process of taking out the ingredients.

[0004] However, the methods of the prior art may make the operation process of food management cumbersome, affecting the user experience. Summary of the invention

[0005] The present application provides an item management method, device and smart refrigerator, which are used to solve the problem in the prior art that the food management process is cumbersome and affects the user experience.

[0006] In a first aspect, the present application provides an item management method, comprising:

[0007] Collecting an image sequence of a user managing items in an item storage device, wherein the images in the image sequence include a user's hand image area and an item image area;

[0008] Tracking the hand image area to obtain the user's management behavior of the item and whether the user performs a code scanning operation on the item, wherein the management behavior includes storing the item in the item storage device or taking the item out of the item storage device;

[0009] Determining the actual type of the item according to whether the user performs a code scanning operation on the item;

[0010] Push prompt information and update the current stored item set of the item storage device, wherein the prompt information is used to indicate the actual type of the item and the management behavior of the user.

[0011] In a second aspect, the present application provides an item management device, comprising:

[0012] a processing module, configured to collect an image sequence of a user managing items in an item storage device, wherein the images in the image sequence include a user's hand image region and an item image region; and

[0013] Tracking the hand image area to obtain the user's management behavior of the item and whether the user performs a code scanning operation on the item, wherein the management behavior includes storing the item in the item storage device or taking the item out of the item storage device; and

[0014] Determining the actual type of the item according to whether the user performs a code scanning operation on the item;

[0015] The push module is used to push prompt information and update the current stored item set of the item storage device, wherein the prompt information is used to indicate the actual type of the item and the management behavior of the user.

[0016] In a third aspect, the present application provides a smart refrigerator, comprising:

[0017] A camera, the camera is located on the top of the smart refrigerator, the camera is used to collect images of the user managing items in the smart refrigerator; a barcode scanner, the barcode scanner is located within the field of view of the camera; and a processor, the processor is used to identify the user's item management behavior and whether the user performs a barcode scanning operation based on the image collected by the camera, and select to identify the actual type of the item based on the image recognition method according to whether the user performs the barcode scanning operation, and / or start the barcode scanner to identify the actual type of the item by barcode recognition, and push prompt information and update the current stored item collection, the prompt information is used to indicate the actual type of the item and the user's management behavior.

[0018] In a fourth aspect, the present application provides an item storage device, comprising:

[0019] A memory for storing program instructions;

[0020] The processor is used to call and execute the program instructions in the memory to perform the method steps described in the first aspect above.

[0021] In a fifth aspect, the present application provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is used to execute the method described in the first aspect above.

[0022] The item management method, device and smart refrigerator provided by the present application can automatically identify the user's management behavior and whether the user performs a code scanning operation by collecting images of the user managing items and performing target tracking. According to whether the user performs a code scanning operation, the actual type of items managed by the user is automatically identified using a corresponding method, and then the information of this item management is pushed to the user and the current stored item collection is updated. In this process, the user can complete the item management and view the results of the item management without performing any additional operations. Therefore, the operation process of the user managing items is greatly simplified, thereby greatly improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1a This is an exemplary front view structural diagram of an article storage device involved in an embodiment of the present application;

[0025] Figure 1b This is an exemplary side view of the structure of the article storage device involved in the embodiment of the present application;

[0026] Figure 2 A flowchart of an item management method provided in an embodiment of the present application;

[0027] Figure 3 A flowchart of an item management method provided in an embodiment of the present application;

[0028] Figure 4 A flowchart for interacting with the user and updating the current set of stored items based on the operation results;

[0029] Figure 5 This is a sample picture of the interface when the deposit is successful;

[0030] Figure 6 This is an example of the interface when the removal fails;

[0031] Figure 7 A flowchart of using the first method to identify the actual type of items when a user deposits items;

[0032] Figure 8 A flowchart of using the first method to identify the actual type of an item when a user takes out an item;

[0033] Fig. 9A flowchart of using the second method to identify the actual type of items when users deposit items;

[0034] Fig.10 A schematic diagram of a process for identifying the actual type of an item using the second method when the user takes out the item;

[0035] Fig.11 A schematic diagram of the process of obtaining management behavior and whether the user performs a code scanning operation by using target tracking;

[0036] Fig.12a This is an example diagram of the recognition of the removal action;

[0037] Figure 12b This is an example diagram of the recognition of Huizi action;

[0038] Fig.12c This is an example diagram for the recognition of the stay sub-action;

[0039] Fig.13 A module structure diagram of an item management device provided in an embodiment of the present application;

[0040] Fig.14 A schematic diagram of the structure of an item storage device 1400 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] The embodiment of the present application is applied to the scenario where the user uses the item storage device to manage items. The item storage device can be, for example, a smart refrigerator, a smart freezer, etc. The user manages items including the user depositing items into the item storage device and the user taking items out of the item storage device. In the embodiment of the present application, when the user manages items, the item storage device uses a camera installed on the item storage device to collect images of the user when performing deposit or withdrawal operations, and uses the method of the embodiment of the present application to select image recognition-based item management, and / or start the barcode scanner on the item storage device to manage items in a barcode recognition manner. Specifically, it includes: when the user deposits an item, the identified item information is pushed to the user, and the current existing item collection is updated after the user confirms; and when the user takes out an item, it is determined whether the result of the withdrawal operation is successful, and the corresponding prompt information is pushed, and then the current existing item collection is updated according to the user's response information.

[0043] Figure 1a This is an exemplary front view structural diagram of an article storage device involved in an embodiment of the present application. Figure 1b This is an exemplary side view of the structure of the article storage device involved in the embodiment of the present application, such as Figure 1a and Figure 1b As shown, taking the item storage device as a smart refrigerator as an example, a camera is set on the top of the smart refrigerator, and the field of view of the camera faces downward. The barcode scanner is located on the box between the upper and lower refrigeration chambers and the variable temperature chamber. In addition, the barcode scanner can also be located on the outside of the smart refrigerator, on the door body or between the door bodies. The present application does not limit the installation position of the barcode scanner, it is only necessary to make the barcode scanner within the field of view of the camera. The camera can collect images when the user manages items, and the barcode scanner can scan the barcode on the item to obtain the information of the item.

[0044] For ease of description, the following embodiments are described by taking the article storage device as a smart refrigerator as an example. In addition, in some of the following embodiments, the "article storage device" is referred to as "device".

[0045] Figure 2 This is a flow chart of the article management method provided in the embodiment of the present application, and the execution subject of the method is an article storage device. Figure 2 As shown, the method includes:

[0046] S201. Collecting an image sequence of a user managing items in an item storage device, wherein the images in the image sequence include a user's hand image region and an item image region.

[0047] For example, after the user opens the door, the camera on the top of the smart refrigerator can collect multiple frames of images of the user's operation in real time, and the multiple frames of images constitute an image sequence. Each frame of the image sequence contains at least the user's hand and the item held by the user, so that the image contains at least the hand image area and the item image area.

[0048] S202: Track the hand image area to determine the user's management behavior of the item and whether the user performs a code scanning operation on the item. The management behavior includes storing the item in an item storage device or taking the item out of the item storage device.

[0049] Optionally, the image sequence captured by the camera contains the movement information of the user's hands when depositing or withdrawing items. Therefore, by performing target tracking based on the hand image area of ​​each image in the image sequence, it can be known whether the operation performed by the user is depositing or withdrawing, and whether the user has performed a code scanning operation.

[0050] The specific execution process of target tracking in the hand image area will be described in detail in the following embodiments.

[0051] S203: Determine the actual type of the item according to whether the user performs a code scanning operation on the item.

[0052] By tracking the target of the hand image area, the item storage device can automatically identify whether the user has performed a code scanning operation, and based on this information, automatically identify the actual type of the item using a corresponding method. For example, if the user has performed a code scanning operation, the actual type of the item is identified by a barcode recognition method, and if the user has not performed a code scanning operation, the actual type of the item is identified by an image recognition method.

[0053] The actual type of the item may include, for example, brand, specification, category, etc. For example, the actual type of an item is: 200 ml of brand A milk.

[0054] It is worth noting that the above step S202 can be completed before or after step S203 is executed. For example, first, step S202 identifies that the user has performed a code scanning operation, and then step S203 can be started. At the same time, step S202 can continue to be executed to identify the user's management behavior.

[0055] S204: Push prompt information and update the current stored item set of the item storage device. The prompt information is used to indicate the actual type of the item and the user's management behavior.

[0056] Among them, the current stored item set of the item storage device includes various items currently stored in the device. The item storage device can display the items in the current stored item set to the user according to the user's needs, so that the user can know at any time whether certain items need to be replenished.

[0057] In this step, after the item storage device recognizes the user's management behavior and the types of items managed by the user this time, on the one hand, it can push the management behavior and the types of items to the user so that the user knows that the item storage device has recognized this operation. On the other hand, the item storage device also updates the current stored item set according to this operation so that the current stored item set is consistent with the items actually stored in the device, that is, to ensure the accuracy of the current stored item set. Exemplarily, when a user deposits an item A into a smart refrigerator, the smart refrigerator displays a prompt message "You have deposited item A" on its display screen and adds the information of item A to the current stored item set. When a user takes out an item B from the smart refrigerator, the smart refrigerator displays a prompt message "You have taken out item B" on its display screen and deletes the information of item B from the current stored item set.

[0058] In this embodiment, the item storage device can automatically identify the user's management behavior and whether the user performs a code scanning operation by collecting images of the user when managing items and performing target tracking. According to whether the user performs a code scanning operation, the actual type of items managed by the user is automatically identified using a corresponding method. The item storage device then pushes the information of this item management to the user and updates the current stored item collection. In this process, the user can complete the item management and view the results of the item management without performing any additional operations. Therefore, the operation process of the user managing items is greatly simplified, thereby greatly improving the user experience.

[0059] When managing items, the user may or may not perform a code scanning operation. If the user performs a code scanning operation, the item storage device may obtain the actual type of the item by using a barcode recognition method. If the user does not perform a code scanning operation, the actual type of the item may be obtained by using an image recognition method. The following two scenarios may occur during actual implementation: In the first scenario, the user performs a code scanning operation when depositing an item, but does not perform a code scanning operation when taking out the item. In the second scenario, the user does not perform a code scanning operation when depositing an item, but performs a code scanning operation when taking out the item. In the above two scenarios, the item storage device uses different recognition methods when the user deposits and takes out items, and different recognition methods may have different recognition results for the same food. When the user takes out an item, the item storage device may fail to match the identified item category in the current stored item set when updating the current stored item set. Considering this problem, as an optional implementation method, when the item storage device performs the above step S204, before updating the current stored item set, it can first determine whether the operation result of the user's management behavior on the item is a successful operation or an operation failure. Among them, for the user's deposit operation, a successful operation means that the information of the item can be successfully added to the current stored item set when the item is deposited. For the user's withdrawal operation, a successful operation means that the actual category of the identified item can be found from the current stored item set, and a failed operation means that the actual category of the identified item can be found from the current stored item set. For the deposit operation, as long as the item category is identified, it can be added to the current stored item set. Therefore, the operation result when depositing can be directly considered as a successful operation. For the withdrawal operation, the operation result can be judged according to the actual type of the item and the current stored item set through the following item association reasoning process. Specifically, if the current stored item set includes an item that matches the identified item, the operation is determined to be successful.

[0060] Figure 3 A flowchart of the article management method provided in the embodiment of the present application is shown in FIG. Figure 3As shown, the associative reasoning process includes:

[0061] S301, query whether there is an item of the same type as the identified actual type in the current stored item set, if yes, execute step 303, if not, execute step S302.

[0062] For example, the above-mentioned same type of items may refer to the same brand, specification, and category. For example, if the actual type identified is brand A's milk with a specification of 200 ml, and there is an item of brand A's milk with a specification of 200 ml in the currently stored items, then it is determined that the operation is successful, that is, the retrieval is successful.

[0063] S302, query whether there is an item in the current stored item set whose type is different from the actual type identified by less than a preset threshold. If so, execute step S303; if not, execute step S304.

[0064] Exemplarily, the preset threshold may refer to one of brand, specification, and category.

[0065] The difference between the types in the stored item set and the actual types identified is less than a preset threshold, for example, can be:

[0066] (1) Same brand, different specifications, same category.

[0067] For example, if there is a 300ml cola of brand A in the stored item set, and the actual type identified is a 250ml cola of brand A, it is considered that the difference between the two is less than the preset threshold, and the operation is successful.

[0068] (2) Different brands, same specifications, same category

[0069] For example, if there is a 300ml cola of brand A in the stored item set, and the actual type identified is a 250ml cola of brand A, it is considered that the difference between the two is less than the preset threshold, and the operation is successful.

[0070] In another example, although item A and item B are quite different in category classification, they are similar in appearance. Therefore, it can be considered that the difference in category between the two is less than the preset threshold and the operation is successful.

[0071] S303: Determine whether the operation is successful.

[0072] S304: Determine that the operation fails.

[0073] After determining the operation result of the user's item management behavior as a success or failure through the above process, the system can interact with the user based on the operation result and update the current stored item collection, thereby completing item management.

[0074] Figure 4 A flowchart for interacting with the user and updating the current stored item collection based on the operation results, such as Figure 4 As shown, for the successful results of deposit and withdrawal, i.e., deposit success and withdrawal success, the user can be prompted to confirm the prompt information, and the collection is updated after the user confirms. For the failed results of withdrawal, i.e., withdrawal failure, the user can be prompted to correct the item type, and the collection is updated after the user corrects it.

[0075] Optionally, if the above operation result is successful, the currently stored item set is updated according to the user's confirmation information of the above prompt information. If the above operation result is failed, the actual type of the item is updated according to the user's correction information of the prompt information, and the currently stored items are updated according to the actual type of the item.

[0076] The user's correction information for the prompt information may be information about the actual type of correction identified, or information about the correction of the currently stored item set.

[0077] In one example, when a user deposits item A, he / she performs a scan code operation. The device identifies item A as item A through barcode recognition and adds it to the stored item collection. When the user takes out item A, he / she does not perform a scan code operation. The device identifies item A as item B through image recognition. At this time, the user can correct the identified item B to item A, and the device deletes item A from the current stored item collection based on the item A corrected by the user.

[0078] In another example, when the user deposits item A, he does not perform a scan code operation. The device identifies item A as item B through image recognition and adds it to the stored item collection. When the user takes out item A, he performs a scan code operation. The device identifies item A as item A through barcode recognition. At this time, the user can correct item B recorded in the current stored item collection to item A and delete item A from the current stored item collection.

[0079] When the item storage device interacts with the user, it can be done through the interface interaction of the item storage device, the application interface interaction on the terminal device bound to the item storage device, or voice interaction, etc. The following takes the interface interaction of the item storage device as an example for explanation.

[0080] Figure 5 This is a sample interface diagram when the deposit is successful. Figure 5As shown, the item storage device recognizes that the user has deposited item A. Since it is a deposit, the item storage device directly determines that the operation result is successful, that is, the deposit is successful. Therefore, a prompt message "You have deposited item A, please confirm" is displayed on the screen of the item storage device. The user clicks the OK button below the prompt message to confirm. The item storage device receives the user's confirmation information and then adds item A to the current stored item collection.

[0081] Figure 6 This is an example of the interface when the removal fails. Figure 6 As shown, a certain item is item A, and the information stored when it is stored is "item A". When the user takes out the item, the item storage device recognizes that the user has taken out item B, and item B does not exist in the current stored item set. The item storage device then displays a prompt message "Did you take out item B? If not, please correct it" on the screen of the item storage device. The user clicks the correction button below the prompt message, and an input box is displayed on the screen of the item storage device. The user enters the actual category of the item taken out, that is, the correction information "item A", in the input box, and clicks the confirmation button below. The item storage device deletes item A from the current stored item set according to the correction information entered by the user.

[0082] The following describes the process in which the device determines the actual type of the item according to whether the user performs a code scanning operation on the item in the above step S203.

[0083] The actual type of the item may be determined in any of the following two ways.

[0084] In the first method, the device first determines the target recognition method according to whether the user performs a code scanning operation, and the target recognition method is a barcode recognition method or an image recognition method. Then, the target recognition method is used to identify the actual type of the object.

[0085] This method selects a corresponding recognition method for recognition processing according to whether the user performs a code scanning operation, which can save computing overhead.

[0086] Specifically, if the target recognition method is an image recognition method, the device selects a frame of images in the image sequence, performs image recognition on the object image area of ​​the frame of images, and obtains the actual type of the object. If the target recognition method is a barcode recognition method, a barcode scanner is used to perform barcode recognition, and the result of the barcode recognition is used as the actual type of the object.

[0087] Figure 7 A flowchart of using the first method to identify the actual type of items when a user deposits items, such as Figure 7 As shown, the process includes:

[0088] S701: Identify the user's code scanning operation.

[0089] S702, determine whether the barcode scanner is started successfully, if so, execute step S703, otherwise, execute step S704.

[0090] S703, set the scan code flag to 1.

[0091] S704, set the scan code flag to 0.

[0092] S705: Identify that the user's management behavior is deposit.

[0093] S706: Determine whether the code scanning flag is 1. If so, execute step S707; otherwise, execute step S709.

[0094] S707: Use barcode recognition to identify the actual type of the deposited items.

[0095] S708, set the code scanning flag to 0.

[0096] After obtaining the actual type of the object, the scan code flag is set to 0 for the next recognition process.

[0097] S709: Use image recognition to identify the actual type of the deposited items.

[0098] S710: Get the actual type of the deposited item.

[0099] Figure 8 The following is a flow chart of using the first method to identify the actual type of an item when a user takes out an item. Figure 8 As shown, the process includes:

[0100] S801. Identify that the user's management behavior is removal.

[0101] S802: Identify the user's code scanning operation.

[0102] S803, determine whether the barcode scanner is started successfully, if so, execute step S804, otherwise, execute step S805.

[0103] S804, set the scan code flag to 1.

[0104] S805, set the scan code flag to 0.

[0105] S806: Determine whether the code scanning flag is 1. If so, execute step S807; otherwise, execute step S809.

[0106] S807: Use barcode recognition to identify the actual type of the taken out item.

[0107] S808, set the scan code flag to 0.

[0108] After obtaining the actual type of the object, the scan code flag is set to 0 for the next recognition process.

[0109] S809: Use image recognition to identify the actual type of the taken out object.

[0110] S810: Get the actual type of the taken-out item.

[0111] In the second method, the device first performs image recognition on the object image area of ​​a frame of an image in the image sequence to obtain the optional types of the object, and then determines the actual type of the object based on whether the user performs a code scanning operation and the optional types of the object.

[0112] Specifically, if the user performs a code scanning operation, the optional categories are updated according to the barcode recognition result, and the updated optional categories are used as the actual categories of the item. If the user does not perform a code scanning operation, the optional categories of the item are used as the actual categories of the item.

[0113] In this method, the type is first identified using image recognition at the beginning of the recognition process, which can improve the recognition speed and further improve the user experience.

[0114] Fig. 9 A flowchart of using the second method to identify the actual type of items when users deposit items, such as Fig. 9 As shown, the process includes:

[0115] S901: Identify the user's code scanning operation.

[0116] S902: Use image recognition to identify the optional types of deposited items.

[0117] S903, determine whether the barcode scanner is started successfully, if so, execute step S904, otherwise, execute step S905.

[0118] S904, set the scan code flag to 1.

[0119] S905, set the scan code flag to 0.

[0120] S906: Identify that the user's management behavior is deposit.

[0121] S907, determine whether the code scanning flag is 1, if so, execute step S908, otherwise execute step S911.

[0122] S908: Use barcode recognition to identify.

[0123] S909, set the scan code flag to 0.

[0124] After obtaining the actual type of the object, the scan code flag is set to 0 for the next recognition process.

[0125] S910. Use the result of barcode recognition to update the optional categories and obtain the actual categories.

[0126] S911. Take the optional category as the actual category.

[0127] S912. Get the actual type of the deposited item.

[0128] Fig.10 The flowchart of using the second method to identify the actual type of the item when the user takes out the item is shown in FIG. Fig.10 As shown, the process includes:

[0129] S1001. Identify that the user's management behavior is withdrawal.

[0130] S1002: Use image recognition to identify the available types of items to be taken out.

[0131] S1003: Identify the user's code scanning operation.

[0132] S1004, determine whether the barcode scanner is started successfully, if so, execute step S1005, otherwise, execute step S1006.

[0133] S1005, set the scan code flag to 1.

[0134] S1006, set the scan code flag to 0.

[0135] S1007, determine whether the code scanning flag is 1, if so, execute step S1008, otherwise execute step S1011.

[0136] S1008. Use barcode recognition to identify.

[0137] S1009, set the scan code flag to 0.

[0138] After obtaining the actual type of the object, the scan code flag is set to 0 for the next recognition process.

[0139] S1010. Use the result of barcode recognition to update the optional categories and obtain the actual categories.

[0140] S1011. Take the optional category as the actual category.

[0141] S1012: Get the actual type of the taken-out item.

[0142] In the above two methods, when using barcode recognition to identify the type, based on Figure 1a and Figure 1b The barcode scanners exemplified are barcode scanners that scan to get the type of items. Barcode scanners can be photoelectric couplers (CCD), laser scanners, light pen scanners, etc. Among them, CCD scanners mainly use a fixed light beam (usually a floodlight source of a light-emitting diode) to illuminate the entire barcode, reflect the barcode symbol onto the photosensitive element array, and identify the barcode symbol through photoelectric conversion. Some CCD scanners can not only recognize one-dimensional barcodes and row-type two-dimensional barcodes, but also matrix-type two-dimensional barcodes. Laser scanners are scanners that use lasers as light sources. Laser scanners can scan at long distances and have high scanning accuracy. Light pen scanners are handheld contact barcode readers. When using them, the user needs to touch the light pen to the surface of the barcode. When the light spot emitted by the light pen passes through the barcode from left to right, the light in the "space" part is reflected, and the light in the "bar" part will be absorbed. After photoelectric conversion, the electrical signal is amplified and shaped for use in the decoder.

[0143] When using image recognition to identify types, in one method, feature extractors and classifiers can be used to implement image feature extraction, classifier learning and training, and test recognition. Among them, feature extractors can include Histogram of Oriented Gradient (HOG), Local Binary Pattern (LBP), Deformable Parts Model (DPM), etc. Classifiers can use Support Vector Machines (SVM), Adaboost, decision trees, Bayesian networks, neural networks, etc. In another method, deep learning methods can be used for training and recognition, such as Fast-CNN, Faster-CNN, YOLO, YOLO 9000, etc.

[0144] The following describes the process of performing target tracking on the hand image area in the above step S202 to obtain the user's management behavior on the item and whether the user performs a code scanning operation on the item.

[0145] Specifically, the device tracks the hand image area to obtain the hand movement trajectory, and based on the hand movement trajectory, obtains the user's management behavior and whether the user performs a code scanning operation.

[0146] Fig.11 This is a schematic diagram of the process of using target tracking to obtain management behavior and whether the user performs a code scanning operation, such as Fig.11As shown, first, the device identifies the hand image area in the image. Secondly, it tracks the area and identifies the three sub-actions of go, return and stay according to the hand tracking trajectory. Furthermore, the user's management behavior can be identified based on the go and return sub-actions, and whether the user performs a code scanning operation can be identified based on the stay sub-action.

[0147] When identifying the hand image area in the image, the image area including the hand and the object can be detected. The hand area detection method can be the aforementioned feature extractor and classifier detection method, or the aforementioned deep learning-based method.

[0148] When tracking a target, Kalman filtering, particle filtering and other methods can be used to track the target in the hand image area, and the three sub-actions of going, returning and staying can be identified based on the tracking trajectory. Fig.12a This is an example diagram of the recognition of the sub-action. Figure 12b This is an example diagram of the recognition of Huizi action. Fig.12c This is an example diagram of the recognition of the stay sub-action. Fig.12a As shown in , when the tracking trajectory is from the refrigerator door to the refrigerator cabinet, crossing the boundary between the cabinet and the door, the sub-action is to go, and the management behavior is to store. Figure 12b As shown in , when the tracking trajectory is from the refrigerator body to the refrigerator door, crossing the boundary between the refrigerator body and the door body, the sub-action is return, and the management behavior is take out. Fig.12c As shown, when the tracking trajectory stays in the code scanning stop area for longer than a preset time, the sub-action is determined to be a stop, and then it is determined that the user has performed a code scanning operation.

[0149] Fig.13 A module structure diagram of an item management device provided in an embodiment of the present application, such as Fig.13 As shown, the device comprises:

[0150] The processing module 1301 is used to collect an image sequence when a user manages items in an item storage device, wherein the images in the image sequence include a hand image area of ​​the user and an item image area; and, perform target tracking on the hand image area to obtain the user's management behavior of the items and whether the user performs a code scanning operation on the items, wherein the management behavior includes storing the items in the item storage device or taking the items out of the item storage device; and, determine the actual type of the item based on whether the user performs a code scanning operation on the item.

[0151] The push module 1302 is used to push prompt information and update the current stored item set of the item storage device, wherein the prompt information is used to indicate the actual type of the item and the management behavior of the user.

[0152] As an optional implementation manner, the processing module 1301 is specifically configured to:

[0153] A target recognition method is determined according to whether the user performs a code scanning operation, and the target recognition method includes a barcode recognition method or an image recognition method; the actual type of the object is recognized using the target recognition method.

[0154] As an optional implementation manner, the processing module 1301 is specifically configured to:

[0155] If the target recognition method is an image recognition method, image recognition is performed on the object image region of a frame of the image sequence to obtain the actual type of the object.

[0156] As an optional implementation manner, the processing module 1301 is specifically configured to:

[0157] If the target identification method is a barcode identification method, the result of the barcode identification is used as the actual type of the object.

[0158] As an optional implementation manner, the processing module 1301 is specifically configured to:

[0159] Image recognition is performed on the object image region of a frame of the image sequence to obtain optional types of the object; and the actual type of the object is determined according to whether the user performs a code scanning operation and the optional types of the object.

[0160] As an optional implementation manner, the processing module 1301 is specifically configured to:

[0161] If the user performs a barcode scanning operation, the selectable categories are updated according to the barcode recognition result, and the updated selectable categories are used as the actual categories of the item.

[0162] As an optional implementation manner, the processing module 1301 is specifically configured to:

[0163] If the user does not perform a code scanning operation, the optional type of the item is used as the actual type of the item.

[0164] As an optional implementation, the push module 1302 is specifically used for:

[0165] Pushing the prompt information; if it is determined that the operation result of the user's management behavior on the item is successful, updating the current stored item collection according to the user's confirmation information on the prompt information.

[0166] As an optional implementation, the push module 1302 is specifically used for:

[0167] Push the prompt information; if it is determined that the operation result of the user's management behavior on the item is an operation failure, update the actual type of the item according to the user's correction information to the prompt information, and update the currently stored items according to the actual type of the item.

[0168] As an optional implementation manner, the processing module 1301 is further configured to:

[0169] The operation result of the user's management behavior on the items is determined according to the actual type of the items and the currently stored item set.

[0170] As an optional implementation manner, the processing module 1301 is specifically configured to:

[0171] If the management behavior is to take out the item from the item storage device, and the current stored item set includes items that match the item, then it is determined that the operation result of the user's management behavior on the item is a successful operation.

[0172] The items that match the items include items whose types are the same as the actual types, or items whose types differ from the actual types by less than a preset threshold.

[0173] As an optional implementation manner, the processing module 1301 is specifically configured to:

[0174] The hand image area is tracked to obtain a hand movement trajectory; based on the hand movement trajectory, the user's management behavior and whether the user performs a code scanning operation are obtained.

[0175] The item management device provided in the embodiment of the present application can execute the method steps in the above method embodiment. Its implementation principles and technical effects are similar and will not be repeated here.

[0176] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the determination module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0177] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0178] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0179] The present application also provides a smart refrigerator, including:

[0180] A camera, the camera is located on the top of the smart refrigerator, the camera is used to collect images of the user managing items in the smart refrigerator; a barcode scanner, the barcode scanner is located within the field of view of the camera; and a processor, the processor is used to identify the user's item management behavior and whether the user performs a barcode scanning operation based on the image collected by the camera, and select to identify the actual type of the item based on the image recognition method according to whether the user performs the barcode scanning operation, and / or start the barcode scanner to identify the actual type of the item by barcode recognition, and push prompt information and update the current stored item collection, the prompt information is used to indicate the actual type of the item and the user's management behavior.

[0181] Fig.14 This is a schematic diagram of the structure of an item storage device 1400 provided in an embodiment of the present application. Fig.14As shown, the item storage device may include: a processor 141, a memory 142, a communication interface 143, a system bus 144, a camera 145 and a barcode scanner 146. The memory 142, the communication interface 143, the camera 145 and the barcode scanner 146 are connected to the processor 141 through the system bus 144 and communicate with each other. The memory 142 is used to store computer execution instructions, the communication interface 143 is used to communicate with other devices, and the camera 145 is used to collect images when the user manages items and send them to the processor 141. The barcode scanner 146 is used to scan the barcode on the item to obtain the actual type of the item and send it to the processor 141. When the processor 141 executes the computer program, it realizes the above Figure 2 To the solution of the embodiment shown in Figure 12.

[0182] Should Fig.14 The system bus mentioned in the specification may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as a client, a read-write library, and a read-only library). The memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0183] The above-mentioned processor can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0184] Optionally, the present application also provides a storage medium, wherein the storage medium stores instructions, which, when executed on a computer, cause the computer to execute the above-mentioned Figure 2 To the method of the embodiment shown in Figure 12.

[0185] Optionally, the present application embodiment further provides a chip for executing instructions, the chip being used to execute the above Figure 2 To the method of the embodiment shown in Figure 12.

[0186] The embodiment of the present application also provides a program product, wherein the program product includes a computer program, wherein the computer program is stored in a storage medium, and at least one processor can read the computer program from the storage medium, and when the at least one processor executes the computer program, the above-mentioned Figure 2 To the method of the embodiment shown in Figure 12.

[0187] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0188] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0189] It can be understood that in the embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for managing items, characterized in that: include: Collecting an image sequence of a user managing items in an item storage device, wherein the images in the image sequence include a user's hand image area and an item image area; Tracking the hand image area to obtain the user's management behavior of the item and whether the user performs a code scanning operation on the item, wherein the management behavior includes storing the item in the item storage device or taking the item out of the item storage device; Determining the actual type of the item according to whether the user performs a code scanning operation on the item; Pushing prompt information and updating the current stored item set of the item storage device, wherein the prompt information is used to indicate the actual type of the item and the management behavior of the user; The determining the actual type of the item according to whether the user performs a code scanning operation on the item includes: Determining a target recognition method according to whether the user performs a code scanning operation, wherein the target recognition method includes a barcode recognition method or an image recognition method; Using the target identification method, identifying the actual type of the object; or, Performing image recognition on the object image region of a frame of the image sequence to obtain selectable types of the object; The actual type of the item is determined according to whether the user performs a code scanning operation and the optional types of the item.

2. The method according to claim 1, characterized in that The using the target identification method to identify the actual type of the object includes: If the target recognition method is an image recognition method, image recognition is performed on the object image region of a frame of the image sequence to obtain the actual type of the object; and / or, If the target identification method is a barcode identification method, the result of the barcode identification is used as the actual type of the object.

3. The method according to claim 1, characterized in that The determining the actual type of the item according to whether the user performs a code scanning operation and the optional types of the item includes: If the user performs a barcode scanning operation, the selectable categories are updated according to the barcode recognition result, and the updated selectable categories are used as the actual categories of the item; If the user does not perform a code scanning operation, the optional type of the item is used as the actual type of the item.

4. The method according to any one of claims 1 to 3, characterized in that: The pushing of prompt information and updating of the current stored item set of the item storage device includes: Pushing the prompt information; If it is determined that the operation result of the user's management behavior on the item is a successful operation, the current stored item set is updated according to the user's confirmation information of the prompt information; if it is determined that the operation result of the user's management behavior on the item is a failed operation, the actual type of the item is updated according to the user's correction information of the prompt information, and the current stored items are updated according to the actual type of the item.

5. The method according to claim 4, characterized in that Before updating the actual type of the item according to the correction information of the prompt information by the user, the method further includes: The operation result of the user's management behavior on the items is determined according to the actual type of the items and the currently stored item set.

6. The method according to claim 5, characterized in that The determining, according to the actual type of the item and the currently stored item set, the operation result of the user's management behavior on the item includes: If the management behavior is to take out the item from the item storage device, and the currently stored item set includes an item matching the item, then determining that the operation result of the user's management behavior on the item is a successful operation; The items that match the items include items whose types are the same as the actual types, or items whose types differ from the actual types by less than a preset threshold.

7. The method according to any one of claims 1 to 3, characterized in that: The target tracking of the hand image area to obtain the user's management behavior on the item and whether the user performs a code scanning operation on the item includes: Performing target tracking on the hand image area to obtain a hand movement trajectory; According to the hand movement trajectory, the management behavior of the user and whether the user performs a code scanning operation are obtained.

8. A smart refrigerator, characterized in that: include: A camera, the camera is located on the top of the smart refrigerator, the camera is used to collect an image sequence when the user manages the items in the smart refrigerator, the images in the image sequence include a user's hand image area and an item image area; a barcode scanner, the barcode scanner being located within the field of view of the camera; and a processor, the processor being configured to perform target tracking on the hand image region according to the image captured by the camera, to obtain the user's management behavior on the item and whether the user performs a code scanning operation on the item, the management behavior including storing the item in the item storage device or taking the item out of the item storage device; Identify the user's item management behavior and whether the user performs a code scanning operation, and select to identify the actual type of the item based on an image recognition method according to whether the user performs a code scanning operation, and / or start the barcode scanner to identify the actual type of the item in a barcode recognition method, and push prompt information and update the current stored item collection, wherein the prompt information is used to indicate the actual type of the item and the user's management behavior; The processor is specifically configured to: determine a target recognition method according to whether the user performs a code scanning operation, wherein the target recognition method includes a barcode recognition method or an image recognition method; and use the target recognition method to recognize the actual type of the object; or, Image recognition is performed on the object image region of a frame of the image sequence to obtain optional types of the object; and the actual type of the object is determined according to whether the user performs a code scanning operation and the optional types of the object.

Images