A locker and its management method

Abstract

The present invention discloses a locker and a management method thereof. The first camera module is used to identify the item information of the item to be stored, the second camera module is used to determine the item to be stored, and the pixel coordinates of the item to be stored are determined as the storage position through image positioning. The storage position and the item information are stored correspondingly, and the entry of the item information to be stored and the storage position is completed. During the entry process, there is no need to manually input the item information and manually mark the position information, so the efficiency of the access process is improved and the management of the locker is convenient.

Description

Technical Field

[0001] The present invention relates to the technical fields of digital image processing and machine vision, and particularly to a storage cabinet and a management method thereof. Background Art

[0002] Liquors are generally stored in liquor cabinets. Traditional liquor cabinets are only used for storing liquor collections, suitable for providing an environment for storing liquors, and do not have the functions of inputting liquor information and identifying the storage location. Users cannot clearly know the liquors and their storage locations, and the management of liquors in the liquor cabinet and the process of retrieving designated liquors are very cumbersome. Summary of the Invention

[0003] The purpose of the present invention is to provide a storage cabinet and a management method thereof, which can automatically input the information and storage location of items stored by users, facilitating users to manage and access items.

[0004] An embodiment of the present invention provides a storage cabinet, which includes: a storage cabinet body, a controller, a first camera module installed outside the storage cabinet body, and a second camera module installed inside the storage cabinet body;

[0005] The controller is configured to:

[0006] In response to a control instruction input by a user, collect image data of an item to be stored through the first camera module, and extract the item information of the item to be stored by using image recognition technology;

[0007] Within a preset time after obtaining the image data, obtain a real-time image inside the storage cabinet body through the second camera module, and obtain a candidate image of the item to be stored by using the frame difference method, and identify a suspected area of the item to be stored according to the gray value of the pixels in the candidate image;

[0008] When the suspected area conforms to a feature rule preset for the item information, store the position corresponding to the coordinates of the suspected area in the candidate image in correspondence with the item information.

[0009] Preferably, the extracting of the item information of the item to be stored by using image recognition technology specifically includes:

[0010] Perform convolution filtering processing on the image data, and extract the feature information of the item to be stored in the filtered image data by using optical character recognition technology;

[0011] Query the item information corresponding to the feature information according to a preset item database, and the item information includes the type of the item to be stored.

[0012] As a preferred solution, the second camera module includes at least one second camera unit distributed inside the storage cabinet body. Each second camera unit determines the storage position of the items within the set range it captures by taking images within the set range.

[0013] Preferably, the method for obtaining a real-time image inside the storage cabinet body through the second camera module, obtaining a candidate image of the item through the frame difference method, and identifying a suspected area of the item to be stored based on the gray-scale values of the pixels in the candidate image specifically includes:

[0014] Performing distortion correction on the real-time image obtained by the second camera module, subtracting the background image that has been pre-obtained by the second camera module and subjected to distortion correction from the distortion-corrected real-time image to obtain a frame difference image;

[0015] Performing binary processing on the frame difference image according to a preset gray-scale value threshold, and then performing image closing operation on the obtained image to obtain the candidate image;

[0016] Taking the set of all pixels with the highest gray-scale value in the candidate image as the suspected area.

[0017] Preferably, when the suspected area in the candidate image conforms to the feature rules preset for the item information, storing the position corresponding to the coordinates of the suspected area in the candidate image in correspondence with the item information specifically includes:

[0018] Counting the number of pixels with the highest gray-scale value in the candidate image as the area of the region, and calculating the height and width of the suspected area;

[0019] Calculating the suspected feature of the suspected area based on the area of the region, the height, and the width;

[0020] Querying the feature rules of the item to be stored in the preset feature rule library according to the item information, and judging based on the suspected feature;

[0021] When the suspected feature meets the feature rules, determining that the suspected area is the item to be stored;

[0022] Taking the position corresponding to the coordinates of the pixel with the lowest position in the suspected area as the storage position of the item to be stored, and storing the storage position and the item information in correspondence;

[0023] Wherein, the suspected feature includes the area of the suspected area, the area ratio, and the rectangle ratio; the area ratio is the ratio of the area of the region to the area of the suspected area, and the area of the suspected area is the product of the height and the width; the rectangle ratio is the ratio of the height to the width.

[0024] Further, when the suspected feature meets the feature rule, determining that the suspected area is the item to be stored specifically includes:

[0025] When the area of the area is within the first threshold range preset in the feature rule, and the area ratio is within the second threshold range preset in the feature rule, and the rectangle ratio is within the third threshold range preset in the feature rule, determining that the suspected area is the item to be stored.

[0026] As an alternative to the above solution, when the suspected feature meets the feature rule, determining that the suspected area is the item to be stored specifically includes:

[0027] Query the rectangle ratio score table preset in the feature rule to obtain the corresponding rectangle ratio score of the rectangle ratio;

[0028] Query the area ratio score table preset in the feature rule to obtain the corresponding area ratio score of the area ratio;

[0029] Query the area score table of the area area preset in the feature rule to obtain the corresponding area score of the area area;

[0030] Calculate the sum of the rectangle ratio score, the area ratio score and the area score as the confidence score of the suspected area. When the confidence score is greater than the fourth threshold preset in the feature rule, determine that the suspected area is the item to be stored.

[0031] Preferably, the controller is further configured to:

[0032] Obtain a real-time image inside the locker through the second camera module. When it is recognized by the frame difference method that the item at the position to be taken out in the obtained real-time image is taken out, delete the position to be taken out and the corresponding item information;

[0033] Collect image data of the item to be taken out through the first camera module, extract the item information of the item to be taken out by using image recognition technology, and delete the item information of the item to be taken out and the corresponding position.

[0034] Preferably, the obtaining a real-time image inside the locker through the second camera module. When it is recognized by the frame difference method that the item at the position to be taken out in the obtained real-time image is taken out, deleting the position to be taken out and the corresponding item information specifically includes:

[0035] Subtract the obtained real-time image from the pre-obtained background image to obtain the frame difference image of the item to be taken out;

[0036] By processing the obtained frame difference image, the candidate image is obtained, and the suspected area of the position to be taken out is determined;

[0037] Calculate the suspected features of the suspected area of the position to be taken out. When the calculated suspected features meet the feature rules preset for the item information corresponding to the position to be taken out, it is determined that the item at the position to be taken out has been taken out, and the position to be taken out and the corresponding item information are deleted.

[0038] An embodiment of the present invention further provides a management method for a locker, and the method includes:

[0039] In response to a control instruction input by a user, image data of an item to be stored is collected by a first camera module installed outside the locker body, and item information of the item to be stored is extracted using image recognition technology;

[0040] Within a preset time after obtaining the image data, a real-time image inside the locker body is obtained by a second camera module installed inside the locker body, and a candidate image of the item to be stored is obtained by the frame difference method. The suspected area of the item to be stored is identified according to the gray scale value of the pixels in the candidate image;

[0041] When the suspected area meets the feature rules preset for the item information, the position corresponding to the coordinates of the suspected area in the candidate image is stored corresponding to the item information.

[0042] A locker and its management method provided by the present invention identify the item information of the item to be stored through the first camera module, determine the item to be stored through the second camera module, and determine the coordinates of the pixels of the item to be stored as the storage position through image positioning, and store the storage position and the item information correspondingly, completing the entry of the item information to be stored and the storage position; during the entry process, there is no need for manual input of item information and manual marking of position information, improving the efficiency of the access process and facilitating the management of the locker. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic structural diagram of a locker provided by an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of the work flow executed by the controller in an embodiment of the present invention;

[0045] Figure 3 is a schematic flow diagram of a management method for a locker provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0047] See Figure 1 , which is a schematic structural diagram of a locker provided by an embodiment of the present invention. An embodiment of the present invention provides a locker 10, including: a locker body 11, a first camera module 12, a second camera module 13, and a controller 14.

[0048] The first camera module 12 is installed outside the locker body 11 for the user to obtain image data outside the locker; the second camera module 13 is installed inside the locker body 11 for obtaining image data of the items stored inside the locker.

[0049] The controller 14 controls the first camera module 12 and the second camera module 13 to obtain image data and processes the obtained image data to realize the management of the items stored in the locker.

[0050] See Figure 2 , which is a schematic diagram of the work flow executed by the controller in the embodiment of the present invention. The controller is used to execute steps S1 to S3:

[0051] S11. In response to a control instruction input by the user, collect image data of the item to be stored through the first camera module, and extract the item information of the item to be stored by using image recognition technology;

[0052] S12. Within a preset time after obtaining the image data, obtain a real-time image inside the locker body through the second camera module, and obtain a candidate image of the item to be stored by using the frame difference method, and identify a suspected area of the item to be stored according to the gray value of the pixels in the candidate image;

[0053] S13. When the suspected area conforms to the feature rule preset for the item information, store the position corresponding to the coordinate of the suspected area in the candidate image and the item information correspondingly.

[0054] In the specific implementation of this embodiment, after the user inputs a control instruction to store wine, the controller controls the first camera module to start working. The first camera module collects multiple frames of images at a fixed angle in real time. The obtained images contain the image of the item to be stored. Through image recognition technology, the item information of the item to be stored can be recognized;

[0055] It should be noted that when the first camera module recognizes an image, the shooting angle of the first camera module can be set to the front or other set angles, and the user needs to place the item to be stored at a specified position for recognition; the first camera module automatically recognizes the item to be stored within the viewing angle and adjusts the shooting angle to obtain the image data of the item to be stored.

[0056] The specific implementation methods of the process of the user inputting a control instruction include the following several types:

[0057] By automatically sensing the user's storage behavior, input the control instruction: Detect the distance between the user and the locker through detection units such as infrared cameras or radars. When the distance between the user and the locker is less than the first preset value, when the user is in front of the locker; when it is detected that the user is in front of the locker, start calculating the user's stay time. When the distance between the user and the locker is greater than the first preset value, when the user leaves the locker, the stay time is cleared; by detecting the user's stay time, input the control instruction, and through automatically sensing the user's storage behavior, realize the braking interaction, improve the user's experience, and avoid accidental touch caused by the user passing by the locker by detecting the stay time.

[0058] By the set interaction unit, detect the user's input instruction: Through the touch screen or buttons set on the locker surface, the user actively triggers a specific button to input the control instruction;

[0059] By collaborative control, input the control instruction: By detecting the user's behavior of opening the storage cabinet door, synchronously input the control instruction.

[0060] After the first camera module obtains the image data, it indicates that the user is about to deposit an item. At this time, within a preset time period, when it is detected that the user deposits an item, the item deposited by the user is corresponded with the item information recognized through the image data, indicating that the item to be stored is stored in the corresponding position. Specifically:

[0061] Within the preset time after obtaining the image data, detect the change of the image obtained by the second camera module before and after the user deposits the item to be stored into the locker. Obtain the candidate image of the item to be stored through the frame difference method, and based on the gray value of the pixels in the candidate image, regard the pixels with a large difference in gray value from the background as the suspected area of the item to be stored; adopt the frame difference method to eliminate the interference of the background part in the locker and obtain the suspected area where the item to be stored is placed;

[0062] By detecting the characteristic rules of the suspected area, determine whether the suspected area conforms to the characteristic rules corresponding to the item information. Different items correspond to different characteristic rules, specifically information such as the size of the item and the aspect ratio of the item's length and width, and determine that the suspected area is the item to be stored, improving the accuracy of item position recognition.

[0063] When it is determined that the suspected area is an item to be stored, based on the coordinate position of the suspected area in the candidate image and the shooting angle of the second camera module, the specific position of the item to be stored in the locker can be identified.

[0064] The item information of the item to be stored is identified by the first camera module, the item to be stored is determined by the second camera module, and through image positioning, the coordinates of the pixels of the item to be stored are determined as the storage position. The storage position and the item information are stored correspondingly to complete the entry of the information of the item to be stored and its storage position. During the entry process, there is no need for manual input of item information and manual marking of position information, which improves the efficiency of the access process and facilitates the management of the locker.

[0065] In another embodiment provided by the present invention, the extraction of the item information of the item to be stored by using the image recognition technology specifically includes:

[0066] Perform convolution filtering processing on the image data, and extract the characteristic information of the item to be stored in the filtered image data through optical character recognition technology;

[0067] Query the item information corresponding to the characteristic information according to a preset item database, and the item information includes the type of the item to be stored.

[0068] In this embodiment, since the image data obtained by the first camera module is obtained during the process of the user depositing an item, during the process of the user depositing an item, the item to be stored is in a moving state, and the image data obtained at this time has motion blur. Through convolution filtering processing, the motion blur can be eliminated, the clarity of the image data can be improved, and the accuracy of item information recognition can be improved.

[0069] The image data obtained is generally the text information of the item to be stored. Specifically, for example, when storing wine, when the user stores wine, the barcode information or the text information of the label of the wine is obtained as the characteristic information, and then the item information corresponding to the characteristic information is queried by querying the networked database to identify the item information of the item to be stored; for example, the item information is obtained through the barcode, or the item information is queried through the label name of the commodity.

[0070] By obtaining image data, through optical character recognition technology, that is, OCR recognition technology, the characters in the image are detected by detecting the dark and bright patterns to determine their shapes, and then the shapes are translated into computer text by character recognition methods; that is, for printed characters, the text in the paper document is converted into a black and white dot matrix image file in an optical way, and the text in the image is converted into text format characteristic information by the recognition software.

[0071] Query the item information corresponding to the feature information according to a preset item database. The item information includes the item type, storage environment, etc. In subsequent feature recognition of the item type by the user, determine the features of the suspected image to achieve item recognition.

[0072] The item database can be a correspondence table of basic information and feature information searched through the Internet, or a correspondence table of basic information and feature information established in advance by big data. For example, obtain the label of the wine product as the feature information, and obtain the wine product type corresponding to the label through big data.

[0073] It should be noted that in this embodiment, in the process of querying the item information by identifying the feature information through optical character recognition technology, in other embodiments, the item information can also be directly identified through optical character technology.

[0074] Identify the feature information of the item to be stored through optical character recognition technology, determine the item information according to the feature information, and automatically realize the entry of the item information.

[0075] In another embodiment provided by the present invention, the second camera module includes at least one second camera unit distributed inside the locker body. Each second camera unit determines the storage position of the item within the set range by taking images within the set range.

[0076] In the specific implementation of this embodiment, the second camera module includes several different camera units. Each camera unit determines the storage position of the item within its shooting range by taking images within the set range.

[0077] Specifically, if the locker body has multiple storage layers, a wide-angle camera can be configured for each storage layer. Each wide-angle camera is used to take real-time images of the entire storage layer range. Determine which specific position on which layer the item to be stored is located through the images obtained by different cameras, which can adapt to different locker structures and improve the accuracy of the storage position of the items in the locker.

[0078] In another embodiment provided by the present invention, obtaining the real-time image inside the locker body through the second camera module, and obtaining the candidate image of the item through the frame difference method, and identifying the suspected area of the item to be stored according to the gray value of the pixels in the candidate image specifically includes:

[0079] Perform distortion correction on the real-time image obtained by the second camera module, and subtract the background image that has been pre-obtained by the second camera module and subjected to distortion correction from the distortion-corrected real-time image to obtain a frame difference image.

[0080] Perform binary processing on the frame difference image according to a preset gray value threshold, and then perform image closing operation on the obtained image to obtain the candidate image;

[0081] Take the set of all pixels with the highest gray value in the candidate image as the suspected area.

[0082] In this embodiment, when obtaining the candidate image, since the second camera module generally uses a wide-angle camera to identify images at close range, the wide-angle camera will produce distortion in the image edge area. Therefore, when identifying the specific storage location through pixel coordinates, there is a large error. Therefore, it is necessary to perform distortion correction on the actual image obtained by the second camera module to make the image correspond to the real coordinates of the item in the locker. Through distortion correction, the image quality can also be improved, and the subsequent feature recognition accuracy can be improved.

[0083] Before the item to be stored is placed in the locker, the second camera module needs to obtain the background image in the locker and perform the same distortion correction process on the background image. After performing distortion correction on the real-time image of the item to be stored when it is placed in the locker, subtract it from the corrected background image to obtain the frame difference image after frame difference method processing; the determination criterion for the real-time image can be that when the number of pixel points where the image obtained by the second camera module changes exceeds the preset number, the image obtained in real time is taken as the real-time image. The image obtained by the second camera module changes greatly, indicating that the item to be stored may be placed in the locker.

[0084] Performing binary processing on the frame difference image according to a preset gray value threshold to obtain a binary image specifically:

[0085] The frame difference image is an image with the gray value of image pixels ranging from 0 to 255. By setting a gray value threshold and traversing all the image pixels, the gray value of the pixels not less than the gray value threshold is set to 255, and the gray value of the pixels less than the gray value threshold is set to 0. For example, when the gray value threshold is set to 128, the pixels not less than 128 are 255, and those less than 128 are set to 0. The obtained binary image has only these two gray values, where 0 represents pure black and 255 represents pure white.

[0086] Perform closing operation on the binary image. The specific process of the closing operation is to first perform dilation operation on the binary image and then perform erosion operation to smooth the contour of the object, disconnect the narrow necks and eliminate the thin protrusions, and enclose two slightly connected blocks together. For example, when the item to be stored is a wine product, since the wine bottle is a relatively large and complete object, perform closing operation on the binary image after the frame difference method to form a whole at the small fracture, so as to complete the whole wine bottle and facilitate later feature judgment.

[0087] The frame difference image is binarized to obtain a binary image. After performing a closing operation, the area of the pure white part in the obtained candidate image is the area with a relatively high difference from the background, that is, the suspected area where the item to be stored is placed.

[0088] By performing frame difference processing on the image obtained by the second camera module, followed by binarization processing and closing operation processing, it is possible to eliminate the noise and image interference in the obtained candidate image, facilitating the identification of the suspected area in the image for subsequent feature recognition.

[0089] In another embodiment provided by the present invention, the steps S3 executed by the controller specifically include:

[0090] Count the number of pixels with the highest gray value in the candidate image as the area of the region, and calculate the height and width of the suspected area;

[0091] Calculate the suspected feature of the suspected area according to the area of the region, the height, and the width;

[0092] Query the feature rule of the item to be stored in the preset feature rule library according to the item information, and judge according to the suspected feature;

[0093] When the suspected feature meets the feature rule, determine that the suspected area is the item to be stored;

[0094] Take the position corresponding to the coordinate of the pixel with the lowest position in the suspected area as the storage position of the item to be stored, and store the storage position and the item information correspondingly;

[0095] Among them, the suspected feature includes the area of the suspected area, the area ratio, and the rectangle ratio; the area ratio is the ratio of the area of the region to the area of the suspected area, and the area of the suspected area is the product of the height and the width; the rectangle ratio is the ratio of the height to the width.

[0096] When specifically implementing this embodiment, count the number of pixels with the highest gray value in the candidate image as the area of the region S, and the area of the region represents the proportion of the suspected area in the candidate image;

[0097] Calculate the height and width of the suspected area. The height H is the distance between the highest pixel and the lowest pixel, and the width W is the distance between the leftmost pixel and the rightmost pixel;

[0098] Calculate the suspected feature of the suspected area according to the area of the region, the height, and the width; the suspected feature includes the area of the suspected area, the area ratio, and the rectangle ratio.

[0099] The area ratio Q = S / (W * H) * 100;

[0100] The rectangular ratio R = H / W * 100;

[0101] The area S of the region is the number of pixels with the highest grayscale value in the candidate image.

[0102] Query the feature rules corresponding to the item information in the preset feature rule library through the item information, judge whether the suspected feature meets the feature rules through the feature rules, and judge whether the suspected region is the item corresponding to the item information; the feature rule library includes the feature rules of the item types of different item information obtained in advance through big data, or the feature rules corresponding to the item types of different item information input manually;

[0103] When the suspected feature meets the feature rules, determine that the suspected region is the item to be stored; use the position corresponding to the coordinates of the pixel with the lowest position in the suspected region as the storage position of the item to be stored, and store the storage position and the item information correspondingly;

[0104] When the suspected feature does not meet the feature rules preset for the item corresponding to the item information, determine that the suspected region is not the item to be stored.

[0105] In another embodiment provided by the present invention, when the suspected feature meets the feature rules, determining that the suspected region is the item to be stored specifically includes:

[0106] When the area of the region is within the first threshold range preset in the feature rules, and the area ratio is within the second threshold range preset in the feature rules, and the rectangular ratio is within the third threshold range preset in the feature rules, determine that the suspected region is the item to be stored.

[0107] In this embodiment, as a method for judging suspected features, different threshold conditions are set for different suspected features of different items; whether the suspected region is the item corresponding to the item information is judged by whether each feature of the suspected feature meets the threshold corresponding to the item information.

[0108] For example, when the item to be stored is wine, the first threshold range set for the area of the wine region is [40, +∞), the second threshold range set for the area ratio of the wine is [30, +∞), and the third area of the rectangular ratio of the wine is [150, 600]; when the area of the region is less than 40, it is considered that the suspected region is noise rather than a wine bottle, and if the area ratio is less than 30, it is considered that the suspected region is an interference object; when the rectangular ratio is less than 150 or greater than 600, it is considered that it does not conform to the wine characteristics;

[0109] When the area of the region in the suspected feature is greater than 40, and the area ratio is greater than 30, and the rectangle ratio is within the range of [150, 600], the suspected region is determined to be an item to be stored;

[0110] By determining whether the area, area ratio, and rectangle ratio of the suspected region meet the set threshold conditions, when it is determined that all features are satisfied, the suspected region is determined to be an item to be stored, and this judgment method has a higher accuracy.

[0111] It should be noted that the first threshold range of the area, the second threshold range of the rectangle ratio, and the third threshold range of the area ratio preset for different items are all different. It is necessary to obtain the first threshold range, the second threshold range, and the third threshold range of the item to be stored through the item information obtained by the first camera module, and then make a judgment on the item to be stored.

[0112] In another embodiment provided by the present invention, when the suspected feature meets the feature rule, determining the suspected region as the item to be stored specifically includes:

[0113] Query the rectangle ratio score table preset by the feature rule to obtain the corresponding rectangle ratio score of the rectangle ratio;

[0114] Query the area ratio score table preset by the feature rule to obtain the corresponding area ratio score of the area ratio;

[0115] Query the area score table preset by the feature rule to obtain the corresponding area score of the area;

[0116] Calculate the sum of the rectangle ratio score, the area ratio score, and the area score as the confidence score of the suspected region. When the confidence score is greater than the fourth threshold preset in the feature rule, determine the suspected region as the item to be stored.

[0117] In this embodiment, as another method for judging suspected features, by setting different score tables for different suspected features of different items, scoring each feature of the suspected features, and determining whether the total score of the suspected features meets the threshold, to judge whether the suspected region is the item corresponding to the item information.

[0118] Specifically, when judging the suspected features of wine products, query the rectangle ratio score table preset for wine products to obtain the corresponding rectangle ratio score of the rectangle ratio of the suspected features;

[0119] Table 1 Rectangle Ratio Score Table of Wine Products

[0120] Rectangle ratio R Rectangle ratio score 400≦R<600 40 300≦R<400 50 200≦R<300 30 150≦R<200 20

[0121] Query the area ratio score table preset for wine products to obtain the corresponding area ratio score of the area ratio of the suspected features;

[0122] Table 2 Area ratio score table of wine products

[0123] Area ratio Q Area ratio score 65≦Q 50 60≦Q<65 40 50≦Q<60 30 40≦Q<50 20

[0124] Query the area score table of the preset area of the wine product to obtain the area score corresponding to the area of the suspected feature;

[0125] Table 2 Area score table of wine product area

[0126] Region area S Region area score S<40 10 40≦Q<60 40 60≦Q<80 30 80≦Q 20

[0127] Calculate the sum of the rectangle ratio score, the area ratio score and the area score of the area as the confidence score of the suspected area. When the confidence score is greater than 80, determine that the suspected area is the item to be stored.

[0128] Determine the confidence score through the comprehensive scores of the area, area ratio, and rectangle ratio. When the confidence score is greater than the set threshold, determine the suspected area as the item to be stored. This judgment method has a higher error tolerance.

[0129] It should be noted that the preset area score tables, rectangle ratio score tables, and area ratio score tables of different items are all different. It is necessary to obtain the area score table, rectangle ratio score table, and area ratio score table of the item to be stored through the item information obtained by the first camera module, and then determine the area score, rectangle ratio score, and area ratio score.

[0130] It should be noted that the methods for judging suspected areas through the threshold range and the score table provided by the present invention are two parallel embodiments for judging suspected areas. In other embodiments, the two embodiments can be comprehensively used to judge suspected areas. Specifically:

[0131] When the suspected feature meets the first preset condition and the second preset condition, determine that the suspected area is the item to be stored;

[0132] The first preset condition is: the area is within the first threshold range preset in the feature rule of the item information, the area ratio is within the second threshold range preset in the feature rule of the item information, and the rectangle ratio is within the third threshold range preset in the feature rule of the item information;

[0133] The second preset condition is: query the rectangle ratio score table preset by the feature rule to obtain the corresponding rectangle ratio score of the rectangle ratio; query the area ratio score table preset by the feature rule to obtain the corresponding area ratio score of the area ratio; query the area score table of the region preset by the feature rule to obtain the corresponding area score of the region area; calculate the sum of the rectangle ratio score, the area ratio score, and the area score of the region area as the confidence score of the suspected region, and the confidence score is greater than the fourth threshold preset in the feature rule of the item information.

[0134] In another embodiment provided by the present invention, the controller is further configured to:

[0135] Obtain a real-time image inside the locker through the second camera module, and when it is recognized by the frame difference method that the item at the to-be-taken-out position in the obtained real-time image is taken out, delete the to-be-taken-out position and the corresponding item information;

[0136] Collect image data of the to-be-taken-out item through the first camera module, extract the item information of the to-be-taken-out item using image recognition technology, and delete the item information of the to-be-taken-out item and the corresponding position.

[0137] In the specific implementation of this embodiment, during the operation of the locker, when an item taking-out instruction input by the user is recognized, as long as the taking-out information of the taken-out item is detected in the first camera module or the second camera module, the taking-out information includes the to-be-taken-out item or the to-be-taken-out position, then update the stored item information and storage location. The first camera uses image recognition technology for the collected image data to obtain the item information of the to-be-taken-out item, and the second camera module uses the frame difference method for the collected real-time image to determine the taking-out position.

[0138] By identifying the taken-out item and updating the stored information in real time, it is convenient for the management of the locker.

[0139] In another embodiment provided by the present invention, when obtaining a real-time image inside the locker through the second camera module and deleting the to-be-taken-out position and the corresponding item information when it is recognized by the frame difference method that the item at the to-be-taken-out position in the obtained real-time image is taken out, it specifically includes:

[0140] Subtract the obtained real-time image from the pre-obtained background image to obtain the frame difference image of the to-be-taken-out item;

[0141] Process the obtained frame difference image to obtain the candidate image and determine the suspected region of the to-be-taken-out position;

[0142] Calculate the suspected features of the suspected area at the position to be retrieved. When the calculated suspected features meet the feature rules preset for the item information corresponding to the position to be retrieved, it is determined that the item at the position to be retrieved has been taken out, and the position to be retrieved and the corresponding item information are deleted.

[0143] In the specific implementation of this embodiment, the second camera module needs to obtain the background image in the locker. At this time, the background image includes the item to be retrieved, and the same distortion correction process is performed on the background image. The real-time image of the item to be retrieved taken out from the locker is subjected to distortion correction and then subtracted from the corrected background image to obtain the frame difference image after frame difference method processing. The determination criterion for the real-time image can be that when the number of pixel points where the image obtained by the second camera module changes exceeds the preset number, the image obtained in real time is taken as the real-time image. The image obtained by the second camera module changes greatly, indicating that the item in the locker may have been taken out.

[0144] Perform binary processing on the frame difference image according to the preset gray value threshold to obtain a binary image, specifically:

[0145] The frame difference image is an image with pixel gray values ranging from 0 to 255. By setting a gray value threshold and traversing all the pixels of the entire image, the gray values of the pixels not less than the gray value threshold are set to 255, and the gray values of the pixels less than the gray value threshold are set to 0. For example, if the gray value threshold is set to 128, the pixels not less than 128 are 255, and those less than 128 are set to 0. The obtained binary image has only these two gray values, where 0 represents pure black and 255 represents pure white.

[0146] Perform closing operation on the binary image. The specific process of the closing operation is to first perform dilation operation on the binary image and then perform erosion operation to smooth the contour of the object, disconnect the narrow necks, eliminate the thin protrusions, and enclose two slightly connected patches together. For example, when the item to be retrieved is a wine product, since the wine bottle is a relatively large and complete object, perform closing operation on the binary image after frame difference method to form a whole at the small fracture, thus perfecting the whole wine bottle for subsequent feature judgment.

[0147] After the frame difference image is subjected to binary processing to obtain a binary image and then closing operation, the area of the pure white part in the obtained candidate image is the area with a high difference from the background, that is, the suspected area where the wine product is suspected to be placed.

[0148] By performing frame difference processing, then binary processing and closing operation on the image obtained by the second camera module, the noise and image interference in the obtained candidate image can be eliminated, which is convenient for identifying the suspected area in the image and performing subsequent feature recognition.

[0149] Count the number of pixels with the highest grayscale value in the candidate image as the area S of the region, and the area of the region represents the proportion of the suspected region in the candidate image;

[0150] Calculate the height and width of the suspected region. The height H is the distance between the highest pixel and the lowest pixel, and the width W is the distance between the leftmost pixel and the rightmost pixel;

[0151] Calculate the suspected feature of the suspected region according to the area of the region, the height and the width; the suspected feature includes the area of the suspected region, the area ratio and the rectangle ratio.

[0152] The area ratio Q = S / (W * H) * 100;

[0153] The rectangle ratio R = H / W * 100;

[0154] The area S of the region is the number of pixels with the highest grayscale value in the candidate image.

[0155] Retrieve the feature rules corresponding to the item information of the item through item information query, and judge whether the suspected feature meets the feature rules through the feature rules, and judge that the item taken out from the suspected region is the item corresponding to the item information.

[0156] When the suspected feature meets the feature rules preset for the item corresponding to the retrieved item information, determine that the suspected region is the item to be retrieved; take the position corresponding to the coordinate of the lowest pixel in the suspected region as the storage position of the item to be retrieved, and delete the storage position and the item information corresponding to the retrieved item;

[0157] When the suspected feature does not meet the feature rules preset for the item corresponding to the retrieved item information, determine that the item in the suspected region has not been retrieved.

[0158] It should be noted that in this embodiment, the suspected features include the area of the region, the area ratio and the rectangle ratio. In other embodiments, other suspected features can be used for item recognition, such as the straight-line distance and the number of straight-line segments of the edge contour of the suspected region, etc., for identifying different types of items.

[0159] Another embodiment of the present invention provides a management method for a locker, see Figure 3 , which is a schematic flowchart of a management method for a locker provided by an embodiment of the present invention. The management method includes steps S21 to S23:

[0160] S21. In response to a control instruction input by a user, collect image data of an item to be stored through a first camera module installed outside the locker body, and extract the item information of the item to be stored by using image recognition technology;

[0161] S22. Within a preset time after obtaining the image data, obtain a real-time image inside the locker body through a second camera module installed inside the locker body, and obtain a candidate image of the item to be stored by using the frame difference method. Identify a suspected area of the item to be stored according to the gray values of the pixels in the candidate image;

[0162] S23. When the suspected area conforms to the feature rules preset for the item information, store the position corresponding to the coordinates of the suspected area in the candidate image and the item information in a corresponding manner.

[0163] It should be noted that the management method of the locker provided in the embodiment of the present invention is the same as all the process steps executed by a controller of a locker in the above embodiment. Their working principles and beneficial effects correspond one by one, and thus will not be described in detail.

[0164] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above various methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0165] The above is the preferred implementation manner of the present invention. It should be pointed out that for those of ordinary skill in the art in the technical field, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A locker, characterized in that, it includes: a locker body, a controller, a first camera module installed outside the locker body, and a second camera module installed inside the locker body; The controller is configured to: In response to a control instruction input by a user, collect image data of an item to be stored through the first camera module, and extract item information of the item to be stored by using image recognition technology; Within a preset time after obtaining the image data, obtain a real-time image inside the locker body through the second camera module, obtain a candidate image of the item to be stored through frame difference method, and identify a suspected area of the item to be stored according to the gray value of the pixels in the candidate image; When the suspected area conforms to the feature rules preset for the item information, store the position corresponding to the coordinates of the suspected area in the candidate image corresponding to the item information; The obtaining a real-time image inside the locker body through the second camera module, obtaining a candidate image of the item through frame difference method, and identifying a suspected area of the item to be stored according to the gray value of the pixels in the candidate image specifically includes: Performing distortion correction on the real-time image obtained by the second camera module, subtracting the distorted corrected real-time image from the background image obtained and distorted corrected by the second camera module in advance to obtain a frame difference image; Performing binaryzation processing on the frame difference image according to a preset gray value threshold, and then performing image closing operation on the obtained image to obtain the candidate image; Taking the set of all pixels with the highest gray value in the candidate image as the suspected area.

2. The locker according to claim 1, characterized in that, The extracting item information of the item to be stored by using image recognition technology specifically includes: Performing convolution filtering processing on the image data, and extracting feature information of the item to be stored in the filtered image data by using optical character recognition technology; Querying the item information corresponding to the feature information according to a preset item database, and the item information includes the type of the item to be stored.

3. The locker according to claim 1, characterized in that, The second camera module includes at least one second camera unit distributed inside the locker body, and each second camera unit determines the storage position of the item within a set range by photographing an image within the set range.

4. The locker according to claim 1, characterized in that, The storing the position corresponding to the coordinates of the suspected area in the candidate image corresponding to the item information when the suspected area in the candidate image conforms to the feature rules preset for the item information specifically includes: Counting the number of pixels with the highest gray value in the candidate image as the area of the area, and calculating the height and width of the suspected area; Calculating a suspected feature of the suspected area according to the area of the area, the height and the width; Querying the feature rules of the item to be stored in a preset feature rule library according to the item information, and making a judgment by judging the suspected feature; When the suspected feature satisfies the feature rule, determine that the suspected area is the item to be stored; Use the position corresponding to the coordinates of the pixel with the lowest position in the suspected area as the storage position of the item to be stored, and store the storage position and the item information correspondingly; Wherein, the suspected features include the area of the suspected area, the area ratio, and the rectangle ratio; the area ratio is the ratio of the area to the area of the suspected area, and the area of the suspected area is the product of the height and the width; the rectangle ratio is the ratio of the height to the width.

5. The locker according to claim 4, characterized in that, When the suspected feature satisfies the feature rule, determining that the suspected area is the item to be stored specifically includes: When the area is within the first threshold range preset in the feature rule, and the area ratio is within the second threshold range preset in the feature rule, and the rectangle ratio is within the third threshold range preset in the feature rule, determine that the suspected area is the item to be stored.

6. The locker according to claim 4, characterized in that, When the suspected feature satisfies the feature rule, determining that the suspected area is the item to be stored specifically includes: Query the rectangle ratio score table preset in the feature rule to obtain the corresponding rectangle ratio score of the rectangle ratio; Query the area ratio score table preset in the feature rule to obtain the corresponding area ratio score of the area ratio; Query the area score table preset in the feature rule to obtain the corresponding area score of the area; Calculate the sum of the rectangle ratio score, the area ratio score, and the area score as the confidence score of the suspected area. When the confidence score is greater than the fourth threshold preset in the feature rule, determine that the suspected area is the item to be stored.

7. The locker according to claim 1, characterized in that, The controller is further configured to: Obtain a real-time image inside the locker through the second camera module. When it is recognized by the frame difference method that the item at the position to be taken out in the obtained real-time image is taken out, delete the position to be taken out and the corresponding item information; Collect image data of the item to be taken out through the first camera module, extract the item information of the item to be taken out by using image recognition technology, and delete the item information of the item to be taken out and the corresponding position.

8. The locker according to claim 7, characterized in that, When obtaining a real-time image inside the locker through the second camera module and recognizing by the frame difference method that the item at the position to be taken out in the obtained real-time image is taken out, deleting the position to be taken out and the corresponding item information specifically includes: Subtract the obtained real-time image from the pre-obtained background image to obtain a frame difference image of the item to be taken out; Process the obtained frame difference image to obtain the candidate image, and determine the suspected area at the position to be taken out; Calculate the suspected features of the suspected area at the position to be retrieved. When the calculated suspected features meet the feature rules preset for the item information corresponding to the position to be retrieved, determine that the item at the position to be retrieved has been retrieved, and delete the position to be retrieved and the corresponding item information.

9. A management method for a locker Characterized in that The method includes: In response to a control instruction input by a user, collect image data of an item to be stored through a first camera module installed outside the locker body, and extract the item information of the item to be stored by using image recognition technology; Within a preset time after obtaining the image data, obtain a real-time image inside the locker body through a second camera module installed inside the locker body, and obtain a candidate image of the item to be stored by using the frame difference method. Identify the suspected area of the item to be stored according to the gray scale values of the pixels in the candidate image; When the suspected area meets the feature rules preset for the item information, store the position corresponding to the coordinates of the suspected area in the candidate image in correspondence with the item information; The step of obtaining a real-time image inside the locker body through the second camera module, obtaining a candidate image of the item by using the frame difference method, and identifying the suspected area of the item to be stored according to the gray scale values of the pixels in the candidate image specifically includes: Perform distortion correction on the real-time image obtained by the second camera module, subtract the background image that has been pre-obtained by the second camera module and subjected to distortion correction from the distortion-corrected real-time image to obtain a frame difference image; Perform binary processing on the frame difference image according to a preset gray scale value threshold, and then perform image closing operation on the obtained image to obtain the candidate image; Use the set of all pixels with the highest gray scale value in the candidate image as the suspected area.

Images