A commodity identification method for intelligent container

Abstract

The present invention provides a commodity identification method for a smart cabinet, which obtains commodity images taken by a fisheye camera disposed above each shelf layer in the smart cabinet; uses a pre-trained improved Cascade R-CNN network model to detect commodities in the commodity images, and cuts out commodities in the commodity images according to the commodity detection results to obtain commodity target images; determines whether the quantity of commodities in the smart cabinet has changed according to the quantity of the commodity target images; if so, uses an improved Resnet network model to extract commodity features, compares feature values ​​with a preset commodity feature library to determine the commodity category and corresponding commodity quantity in the smart cabinet at the current moment, and updates the commodity information in the commodity database of the smart cabinet. The scheme of the present invention realizes accurate identification of the category of purchased commodities through the method of commodity feature comparison, and can add new commodity features to the commodity feature library for identifying commodity categories, thereby improving the applicability of smart cabinets to new commodities.

Description

Technical Field

[0001] The present invention relates to the field of smart retail technology, and in particular to a commodity identification method of a smart container. Background Art

[0002] In recent years, with the upgrading of the retail industry and the development of mobile payments, the entire traditional retail industry is facing upgrading and transformation. Coupled with the promotion of national policies, the new retail trend has officially arrived, and smart containers are one of the important directions.

[0003] A smart vending machine is a vending machine where customers open the door by scanning a QR code on their mobile phone or using facial recognition to verify their identity. After taking the goods and closing the door, the smart vending machine automatically determines what goods the customer has taken and settles the bill automatically. One method is to use visual recognition to identify the type and quantity of goods taken and settle the bill automatically.

[0004] However, this method realizes settlement based on commodity classification, which requires an algorithm to accurately identify the category of the taken goods before settlement can be performed. When new categories of goods are added to the smart container, it is difficult for the smart container to identify the category of the newly added goods. Summary of the invention

[0005] The present invention provides a commodity identification method for a smart container. The commodities in the smart container are detected, and the commodity category and quantity information at the current moment are determined by feature comparison. The commodity category and quantity information are uploaded to the business layer for commodity settlement. The present invention can not only accurately identify the category of commodities for settlement, but also, when new categories of commodities are added to the smart container, the newly added commodity features can be added to the commodity feature library for identifying the category of the commodities, without the need to retrain the commodity identification model.

[0006] The present invention provides a commodity identification method for a smart container, comprising:

[0007] Obtain commodity images taken from a bird's-eye view by fisheye cameras installed above shelves on each level in the smart container;

[0008] Detecting the product in the product image using a pre-trained improved Cascade R-CNN network model, and intercepting the product in the product image according to the product detection result to obtain a product target image;

[0009] Determining whether the quantity of goods in the smart container has changed according to whether the quantity of the commodity target graph is equal to the quantity of goods in the commodity database of the smart container;

[0010] If yes, then extract commodity features from the commodity target graph using the improved Resnet network model, compare the feature values ​​with the preset commodity feature library to determine the commodity category and the corresponding quantity of commodities in the smart container at the current moment, and update the commodity information in the commodity database of the smart container, wherein the preset commodity feature library includes feature values ​​of commodities of different categories;

[0011] The output layer of the Backbone part of the improved Cascade R-CNN network model adopts a deformable convolutional network. The improved Cascade R-CNN network model adopts an improved NMS algorithm to calculate the prediction box score, and the calculation formula is as follows:

[0012]

[0013] Among them, S i is the prediction box score, M is the current prediction box with the highest score, bi is the prediction box to be processed, N is the score threshold, L ac L is the distance between the top left corner vertex of the current prediction box with the highest score and the prediction box to be processed. a is the diagonal length of the current highest-scoring prediction box;

[0014] The input end of the improved Resnet network model includes a mask convolution layer and a normal convolution layer. The mask convolution layer multiplies the input image with a preset image mask and convolves to obtain a first feature map. The normal convolution layer convolves the input image to obtain a second feature map. The improved Resnet network model concats the first feature map and the second feature map. The improved Resnet network model also includes an IBN module. The improved Resnet network model uses a cross entropy loss function and a Triplet Loss loss function for model optimization.

[0015] In an optional embodiment, before acquiring the commodity images taken by the fisheye cameras disposed above the shelves on each layer in the smart container, the method further includes:

[0016] Collect product images of various categories to build a product database dataset;

[0017] The improved Resnet network model is used to extract the characteristics of each category of goods in the commodity base database data set and calculate the characteristic value of each commodity, and the category and characteristic value of each commodity are associated and stored in a preset commodity feature library.

[0018] In an optional embodiment, before acquiring the commodity images taken by the fisheye cameras disposed above the shelves on each layer in the smart container, the method further includes:

[0019] Collecting a plurality of commodity image sample data; wherein the plurality of commodity image sample data includes commodity images of different categories;

[0020] Performing color and shape annotation on the plurality of commodity image sample data to obtain a training data set;

[0021] The constructed improved Cascade R-CNN network model is trained using the training data set, and the OHEM algorithm is used to screen out sample data of difficult-to-distinguish commodity images for model optimization, thereby obtaining the pre-trained improved Cascade R-CNN network model.

[0022] In an optional embodiment, the method further includes:

[0023] If it is determined that the number of goods in the smart cabinet has decreased, the category and corresponding quantity of goods in the smart cabinet at the current moment are compared with the product information in the product database to determine the category and quantity of the reduced goods, and payment information is generated according to the category and quantity of the reduced goods for the user to pay.

[0024] The present invention provides a commodity recognition method for a smart cabinet, which obtains commodity images taken by a fisheye camera arranged above each shelf in the smart cabinet, wherein the fisheye camera can look down and photograph commodities on the shelves below; uses a pre-trained improved Cascade R-CNN network model to detect commodities in the commodity images to obtain a commodity target map, and determines whether the quantity of commodities in the smart cabinet has changed; if so, uses an improved Resnet network model to extract commodity features from the commodity target map, compares feature values ​​with a preset commodity feature library to determine the commodity category and corresponding commodity quantity in the smart cabinet at the current moment, and updates the commodity information in the commodity database of the smart cabinet, wherein the preset commodity feature library includes feature values ​​of commodities of different categories. Compared with the prior art, this solution detects the goods in the smart cabinet, compares the detected goods features with the preset goods feature library to determine the category and quantity information of the goods at the current moment, and uploads the category and quantity information of the goods to the business layer for settlement of the goods. When the user purchases the goods, the present invention accurately identifies the category of the purchased goods through the method of comparing the goods features for settlement of the goods. Moreover, when a new category of goods is added to the smart cabinet, the new product features can be added to the product feature library for identifying the category of the goods, without the need to retrain the product recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 A schematic diagram of a scenario architecture on which the present disclosure is based;

[0027] Figure 2 A flowchart of a commodity identification method for a smart container provided in an embodiment of the present disclosure;

[0028] Figure 3 A flowchart of another method for identifying goods in a smart container provided by an embodiment of the present disclosure;

[0029] Figure 4 A flowchart of an improved Cascade R-CNN network model training method provided in an embodiment of the present disclosure;

[0030] Figure 5 A flowchart of another method for identifying goods in a smart container provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Smart vending machines can use visual recognition to identify the category and quantity of goods taken away and automatically settle accounts. The main principle is to use a camera to capture images of users purchasing goods, use a trained product recognition model to identify the category and quantity of goods, and then settle accounts based on the category and quantity of goods purchased.

[0033] However, this method is based on commodity classification to achieve settlement, and it is necessary to train the commodity recognition algorithm in advance to identify the commodities. When new categories of commodities are added to the smart container, it is difficult for the smart container to identify the categories of the newly added commodities, and the commodity recognition algorithm must be retrained, which greatly limits the applicability of the smart container to the newly added commodities.

[0034] Figure 1 A schematic diagram of a scenario architecture based on the present disclosure is shown in FIG. Figure 1As shown, a scenario architecture based on which the present disclosure is based may include a smart container 1 , a commodity identification device 2 , and a fisheye camera 3 .

[0035] The commodity identification device 2 is hardware or software that can interact with the fisheye camera 3 via a network, and can be used to execute the commodity identification method described in the following embodiments.

[0036] When the commodity identification device 2 is hardware, it can be an electronic device with computing functions. When the commodity identification device 2 is software, it can be installed in an electronic device with computing functions. The electronic device includes but is not limited to a server, a smart box, a desktop computer, and the like.

[0037] The fisheye camera 3 may be a hardware device integrated in the smart container 1 and capable of photographing a wide range of objects at a close distance.

[0038] In actual scenarios, the commodity recognition device 2 can be a server integrated or installed on the smart cabinet 1. The commodity recognition device 2 can run on the smart cabinet 1. The commodity recognition device 2 can also be integrated or installed in the back-end server that processes food storage and access images to provide commodity recognition services for the smart cabinet 1. The commodity recognition device 2 obtains the commodity image taken by the fisheye camera 3 during the user's purchase of the commodity. The commodity recognition device 2 uses the method shown in the following embodiment to recognize the commodity image taken during the user's purchase of the commodity, and determines the change of the commodity in the smart cabinet 1.

[0039] The following is a further description of a commodity identification method of a smart container provided by this application:

[0040] Figure 2 The following is a flow chart of a method for identifying goods in a smart container provided by an embodiment of the present disclosure. Figure 2 As shown, a commodity identification method of a smart container provided by an embodiment of the present disclosure includes:

[0041] S21, obtaining commodity images taken from a bird's-eye view by a fisheye camera disposed above each shelf in the smart container.

[0042] Among them, fisheye cameras can capture a wide range of product images at close range.

[0043] In this embodiment, since the user may access goods from any shelf on any layer of the smart cabinet when storing or accessing goods inside the smart cabinet, in order to identify the goods accessed by the user, the position of the fisheye camera can capture all the goods on each shelf. Therefore, multiple fisheye cameras are used to capture the images of all the goods on each shelf from a bird's-eye view, which can avoid product recognition errors caused by blind spots.

[0044] S22. Detect the commodity in the commodity image using the pre-trained improved Cascade R-CNN network model, and intercept the commodity in the commodity image according to the commodity detection result to obtain a commodity target map.

[0045] The product detection result includes product detection frame information, and the product target image is a small image of each detected product.

[0046] In this embodiment, the pre-trained improved Cascade R-CNN network model can be used to detect the product image to obtain product detection frame information; the products in the product image are cropped according to the product detection frame information to obtain a small picture of each product, and then the small picture of each product is further used to determine the category of each product.

[0047] In order to improve the accuracy of product detection by the network model, especially for the object deformation problem and target density problem caused by camera distortion, the original Cascade R-CNN network model is improved. The output layer of the Backbone part of the improved Cascade R-CNN network model adopts a deformable convolutional network. The improved Cascade R-CNN network model uses an improved NMS algorithm to calculate the prediction box score. The calculation formula is as follows:

[0048]

[0049] Among them, S i is the prediction box score, M is the current prediction box with the highest score, bi is the prediction box to be processed, N is the score threshold, L ac L is the distance between the top left corner vertex of the current prediction box with the highest score and the prediction box to be processed. a is the diagonal length of the current highest-scoring prediction box;

[0050] The advantage of this improvement is that the deformable convolution will learn the offset in the x and y directions respectively. This offset is learned according to the data. After the offset, it is equivalent to the scalable change of each block of the convolution kernel, thereby changing the range of the receptive field. The receptive field becomes a polygon, which will make the outline of the convolution consistent with the actual object shape, which can largely offset the object deformation caused by camera distortion. In addition, an improved NMS algorithm is proposed for the dense situation of goods in smart cabinets. The original NMS algorithm simply calculates the IOU of similar prediction boxes, retains it when it is lower than the threshold, and sets it to 0 when it is greater than the threshold. This will lead to poor detection of dense targets. Therefore, a weighted NMS algorithm is designed, which not only calculates the IOU of two similar prediction boxes, but also calculates the ratio of the distance between the upper left corner points of the two similar prediction boxes and the diagonal line. The prediction box less than the threshold is still retained, and the prediction box score greater than the threshold is not set to 0, but multiplied by the weight value, which can improve the detection rate of dense objects.

[0051] S23: determining whether the quantity of commodities in the smart container has changed according to whether the quantity of the commodity target images is equal to the quantity of commodities in the commodity database of the smart container.

[0052] The commodity database of the smart container stores commodity categories and corresponding commodity quantities in the smart container.

[0053] In this embodiment, the number of commodity target images is the number of detected commodities. The number of detected commodities is determined according to the number of commodity target images, and the number of detected commodities is compared with the number of commodities in the commodity database of the smart cabinet. If the number of detected commodities is equal to the number of commodities in the commodity database of the smart cabinet, the number of commodities in the smart cabinet has not changed, otherwise the number of commodities in the smart cabinet has changed.

[0054] S24. If yes, extract commodity features from the commodity target graph using the improved Resnet network model, compare feature values ​​with those in a preset commodity feature library to determine the commodity category and corresponding commodity quantity in the smart container at the current moment, and update commodity information in the commodity database of the smart container, wherein the preset commodity feature library includes feature values ​​of commodities of different categories.

[0055] In this embodiment, the improved Resnet network model can be used to extract features from the small image of each commodity, and the feature value of each commodity is compared with the feature value stored in the preset commodity feature library, and the commodity category with the highest feature value similarity is determined as the category of the detected commodity, so as to obtain the commodity category and the corresponding commodity quantity in the smart container at the current moment, and update the commodity information in the commodity database of the smart container according to the commodity category and the corresponding commodity quantity in the smart container at the current moment, thereby realizing intelligent management of the commodities in the smart container.

[0056] In order to enable the network model to extract better product features, the original Resnet network model is improved to address the problem that the captured product target image will contain part of other products due to the dense arrangement of products in smart containers. The input end of the improved Resnet network model includes a mask convolution layer and a normal convolution layer. The mask convolution layer multiplies the input image with a preset image mask and convolves to obtain a first feature map. The normal convolution layer convolves the input image to obtain a second feature map. The improved Resnet network model concats the first feature map and the second feature map. The improved Resnet network model also includes an IBN module. The improved Resnet network model uses a cross entropy loss function and a Triplet Loss loss function for model optimization.

[0057] The benefit of this improvement is that when the product target map is input into the improved Resnet network model, it will copy a copy of the product target map. The two copies of the product target map are sent to the mask convolution layer and the normal convolution layer respectively. The normal convolution layer does not require additional operations, while the mask convolution layer multiplies the product target map with the preset image mask and then performs convolution. The size of the preset image mask is consistent with the length and width of the product target map, which is used to block the edge area of ​​the product target map. Finally, the feature maps output by the mask convolution layer and the normal convolution layer are concat-ed and input into the subsequent network. In addition, the IBN module is added to the improved Resnet network model to improve the domain generalization ability, and the cross entropy loss function and TripletLoss loss function are used for model optimization. This can increase the distance between classes while ensuring correct classification, which is conducive to subsequent feature value comparison.

[0058] This embodiment provides a commodity recognition method for a smart cabinet, which obtains commodity images taken by a fisheye camera configured above each shelf in the smart cabinet, wherein the fisheye camera can look down and photograph commodities on the shelves below; uses a pre-trained improved Cascade R-CNN network model to detect commodities in the commodity image to obtain a commodity target map, and determines whether the quantity of commodities in the smart cabinet has changed; if so, uses an improved Resnet network model to extract commodity features from the commodity target map, compares feature values ​​with a preset commodity feature library to determine the commodity category and corresponding commodity quantity in the smart cabinet at the current moment, and updates the commodity information in the commodity database of the smart cabinet, wherein the preset commodity feature library includes feature values ​​of commodities of different categories. By adopting the technical solution provided by the embodiment of the present disclosure, it is realized that when a user purchases commodities, the category of the purchased commodities can be accurately identified by the method of commodity feature comparison to settle the commodities, and when a new category of commodities is added to the smart cabinet, the newly added commodity features can be added to the commodity feature library for identifying the category of the commodities, without the need to retrain the commodity recognition model, thereby improving the applicability of the smart cabinet to the newly added commodities.

[0059] The preset commodity feature library is constructed using an improved Resnet network model and is used as a comparison library for commodity categories. In an optional embodiment, in the above Figure 2 Based on the examples, Figure 3 A flowchart of another method for identifying goods in a smart container provided by an embodiment of the present disclosure. Figure 3 As shown, in Figure 2 Based on S21 and before, it also includes:

[0060] S31, collect commodity images of various categories to build a commodity database data set;

[0061] S32. Using the improved Resnet network model, extract the features of each category of goods in the commodity base database data set and calculate the feature value of each commodity, and associate the category and feature value of each commodity and store them in a preset commodity feature library.

[0062] The present technical solution provides a specific method for constructing a preset commodity feature library based on the above technical solution. The advantage of such a setting in the present embodiment is that the preset commodity feature library can determine the commodity category by comparing the feature values, without directly identifying the commodity category based on the commodity features, and new commodity categories can be added to the preset commodity feature library at any time.

[0063] In order to detect untrained products, the Cascade R-CNN network model is improved to detect products by their color and shape. Figure 4A flowchart of an improved Cascade R-CNN network model training method provided in an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, including:

[0064] S41, collecting a plurality of commodity image sample data; wherein the plurality of commodity image sample data includes commodity images of different categories;

[0065] S42, performing color and shape annotation on the plurality of commodity image sample data to obtain a training data set;

[0066] S43. Using the training data set to train the constructed improved Cascade R-CNN network model, and using the OHEM algorithm to screen out sample data of difficult-to-distinguish commodity images for model optimization, to obtain the pre-trained improved Cascade R-CNN network model.

[0067] In this embodiment, the training data set is input into the constructed improved Cascade R-CNN network model, and the commodity sample images labeled with color information and shape information are processed by the improved Cascade R-CNN network model to obtain color and shape prediction information; the loss function value is calculated according to the color and shape prediction information and the labeled color information and shape information, and the loss function value is back-propagated to each layer of the improved Cascade R-CNN network model to update the weight parameters of each layer according to the loss function value; the above training steps are repeated until the improved Cascade R-CNN network model converges.

[0068] When the number of goods in the smart cabinet decreases, it means that the user has taken the goods and needs to settle the goods. Figure 5 A flowchart of another method for identifying goods in a smart container provided in an embodiment of the present disclosure, the method further comprising:

[0069] S51. If it is determined that the quantity of goods in the smart cabinet is reduced, the categories and corresponding quantities of goods in the smart cabinet at the current moment are compared with the goods information in the goods database to determine the categories and quantities of the reduced goods, and payment information is generated according to the categories and quantities of the reduced goods for the user to pay.

[0070] In this embodiment, the number of different categories of goods in the smart cabinet at the current moment can be compared with the number of different categories of goods stored in the goods database to determine the categories and quantities of reduced goods, and the user payment fee can be calculated using the unit price of the goods and the categories and quantities of reduced goods, and corresponding payment information can be generated based on the user payment fee, so that the user can complete the goods settlement based on the payment information.

[0071] It should be noted that the payment information can be a payment QR code, payment link, etc.

[0072] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the claims.

Claims

1. A commodity identification method for a smart container, characterized in that: include: Obtain commodity images taken from a bird's-eye view by fisheye cameras installed above shelves on each level in the smart container; Detecting the product in the product image using a pre-trained improved Cascade R-CNN network model, and intercepting the product in the product image according to the product detection result to obtain a product target image; Determining whether the quantity of goods in the smart container has changed according to whether the quantity of the commodity target graph is equal to the quantity of goods in the commodity database of the smart container; If yes, then extract commodity features from the commodity target graph using the improved Resnet network model, compare the feature values ​​with the preset commodity feature library to determine the commodity category and the corresponding quantity of commodities in the smart container at the current moment, and update the commodity information in the commodity database of the smart container, wherein the preset commodity feature library includes feature values ​​of commodities of different categories; The output layer of the Backbone part of the improved Cascade R-CNN network model adopts a deformable convolutional network. The improved Cascade R-CNN network model adopts an improved NMS algorithm to calculate the prediction box score, and the calculation formula is as follows: Among them, S i is the prediction box score, M is the current prediction box with the highest score, bi is the prediction box to be processed, N is the score threshold, L ac L is the distance between the top left corner vertex of the current prediction box with the highest score and the prediction box to be processed. a is the diagonal length of the current highest-scoring prediction box; The input end of the improved Resnet network model includes a mask convolution layer and a normal convolution layer. The mask convolution layer multiplies the input image with a preset image mask and convolves to obtain a first feature map. The normal convolution layer convolves the input image to obtain a second feature map. The improved Resnet network model concats the first feature map and the second feature map. The improved Resnet network model also includes an IBN module. The improved Resnet network model uses a cross entropy loss function and a Triplet Loss loss function for model optimization.

2. The commodity identification method of the smart container according to claim 1, characterized in that: Before obtaining the commodity images taken by the fisheye cameras disposed above the shelves on each layer in the smart container, the method further includes: Collect product images of various categories to build a product database dataset; The improved Resnet network model is used to extract the characteristics of each category of goods in the commodity base database data set and calculate the characteristic value of each commodity, and the category and characteristic value of each commodity are associated and stored in a preset commodity feature library.

3. The commodity identification method of the smart container according to claim 1 or 2, characterized in that: Before obtaining the commodity images taken by the fisheye cameras disposed above the shelves on each layer in the smart container, the method further includes: Collecting a plurality of commodity image sample data; wherein the plurality of commodity image sample data includes commodity images of different categories; Performing color and shape annotation on the plurality of commodity image sample data to obtain a training data set; The constructed improved Cascade R-CNN network model is trained using the training data set, and the OHEM algorithm is used to screen out sample data of difficult-to-distinguish commodity images for model optimization, thereby obtaining the pre-trained improved Cascade R-CNN network model.

4. The commodity identification method of the smart container according to claim 1 or 2, characterized in that: The method further comprises: If it is determined that the number of goods in the smart cabinet has decreased, the category and corresponding quantity of goods in the smart cabinet at the current moment are compared with the product information in the product database to determine the category and quantity of the reduced goods, and payment information is generated according to the category and quantity of the reduced goods for the user to pay.

Images