Commodity identification method and computer readable storage medium

Abstract

The application provides a commodity identification method, comprising the following steps: S101, establishing a prior model according to the display habit of commodities in a store; S102, identifying the displayed commodities through image recognition to obtain a first identification result; and S103, correcting the first identification result according to the prior model to obtain a second identification result. The application establishes a prior model of commodity display through historical display data in a supermarket or a convenience store and / or display requirements in the store, detects or identifies the displayed commodities through image recognition, and corrects the detection or identification result through the prior model, thereby improving the accuracy of commodity identification and providing a necessary condition for the unmanned operation of the supermarket / convenience store.

Description

TECHNICAL FIELD

[0001] The present application relates generally to the field of image recognition, and in particular to a method for identifying displayed goods. BACKGROUND

[0002] Existing goods identification algorithms extract features from goods pictures through models, and then compare the features with a database to find the closest features. The goods ID corresponding to the selected features in the database is the goods identification result.

[0003] Identifying the goods displayed on the shelves from the shelf photos is one of the main applications of goods identification. Existing algorithms independently process the identification of each goods picture without considering the relationship between adjacent goods. Due to the limitations of existing machine learning techniques, the current feature extraction models cannot perfectly distinguish different goods, resulting in possible identification errors of existing algorithms.

[0004] The content of the background section merely represents the knowledge of the inventor and does not necessarily represent the state of the art. SUMMARY

[0005] In view of at least one defect of the prior art, the present application provides a goods identification method, characterized in that it comprises:

[0006] S101: Establishing a prior model according to the display habits of in-store goods;

[0007] S102: Identifying the displayed goods through image recognition to obtain a first identification result;

[0008] S103: Correcting the first identification result according to the prior model to obtain a second identification result.

[0009] According to one aspect of the present application, step S101 further comprises:

[0010] Establishing a probability model according to the probability of adjacent display of any two goods as the prior model.

[0011] According to one aspect of the present application, the display habits of in-store goods include historical display data and / or in-store display requirements.

[0012] According to one aspect of the present application, step S102 further comprises:

[0013] Taking pictures of the goods on the shelves;

[0014] Performing image recognition on the goods pictures to generate the first identification result.

[0015] According to an aspect of the present application, the identification method further comprises:

[0016] performing target detection on the product photo to generate one or more detection boxes;

[0017] extracting features for targets within the detection boxes;

[0018] comparing the features with the database;

[0019] obtaining the first identification result comprising product identification results and corresponding confidence levels.

[0020] According to an aspect of the present application, the first identification result is a probability vector comprising confidence levels and corresponding product identification results arranged from high to low.

[0021] According to an aspect of the present application, step S103 further comprises:

[0022] for display products on the same shelf, generating a confidence matrix according to the first identification result, wherein the row vectors of the confidence matrix are the first identification results of the products displayed on the same shelf in sequence, and the column vectors of the confidence matrix are the probability vectors of the products displayed at the positions;

[0023] calculating the comprehensive confidence levels of some or all combination sequences in the confidence matrix according to the prior model;

[0024] obtaining the second identification result according to the calculation results of the comprehensive confidence levels.

[0025] According to an aspect of the present application, the second identification result is the combination sequence with the highest comprehensive confidence level and the corresponding product identification result.

[0026] According to an aspect of the present application, step S103 is implemented through a bundle search algorithm or a Viterbi algorithm.

[0027] The present application also provides a computer-readable storage medium comprising computer-executable instructions stored thereon, wherein the executable instructions, when executed by a processor, implement the identification method as described above.

[0028] The preferred embodiment of the present application provides a product identification method, which establishes a prior model of product display through historical display data and / or in-store display requirements in supermarkets or convenience stores, detects or identifies display products through image recognition, and corrects the detection or identification results through the prior model. The preferred embodiment of the present application improves the accuracy of product identification and provides a necessary condition for the unmanned operation of supermarkets / convenience stores. BRIEF DESCRIPTION OF DRAWINGS

[0029] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of the specification, illustrate embodiments of the application and are used to explain the application, but are not intended to limit the application. In the drawings:

[0030] Figure 1 A product recognition method of an embodiment of the present application is shown;

[0031] Figure 2 A first recognition result of an embodiment of the present application is shown;

[0032] Figure 3 A process of product recognition using the product recognition method of an embodiment of the present application is shown;

[0033] Figure 4 A confidence matrix of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] Hereinafter, certain exemplary embodiments will only be described simply. As can be recognized by those skilled in the art, the described embodiments can be modified in various different ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are to be considered exemplary in nature and not restrictive.

[0035] In the description of the present application, it is to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", and the like, indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are used only to facilitate description of the present application and simplify the description, and do not indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0036] In the description of the present application, it is necessary to point out that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integrally connected; can be mechanical connection, or electrical connection or can communicate with each other; can be directly connected, or indirectly connected through an intermediate medium, or the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0037] In the present application, unless otherwise explicitly specified and limited, the "upper" or "lower" of the first feature to the second feature can include that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, the "upper", "above" and "on" of the first feature to the second feature includes that the first feature is directly above and obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The "lower", "below" and "under" of the first feature to the second feature includes that the first feature is directly above and obliquely above the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0038] The following disclosure provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and arrangements of specific examples are described in the following. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeatedly refer to numbers and / or letters in different examples, and such repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or arrangements discussed. In addition, the present application provides examples of various specific processes and materials, but those skilled in the art can realize the application of other processes and / or the use of other materials.

[0039] The embodiments of the present application are described below in conjunction with the accompanying drawings, it should be understood that the embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0040] Modern supermarkets, convenience stores are increasingly becoming unmanned, and in unmanned supermarkets, the identification of displayed goods is a necessary condition for various links such as goods checkout, restocking, and inventory. In the prior art, there is a certain error rate in the detection and identification of goods through image recognition. The present application provides a method for improving the accuracy of displayed goods identification, which uses the correlation between adjacent goods as a constraint to improve the overall accuracy of goods identification without changing the feature extraction model.

[0041] According to a preferred embodiment of the present application, as Figure 1As shown, the present application provides a commodity identification method 10, comprising steps S101-S103. Wherein:

[0042] In step S101, according to the display habits of commodities in the store, a prior model is established.

[0043] In the retail industry, there are certain rules for how commodities are displayed on the shelves, for example: the same shelf displays the same category of commodities, or according to the store's display requirements. Preferably, these display habits can be obtained by statistical analysis of relevant historical display data, or directly obtained according to the store's display requirements, and then a prior model is established.

[0044] According to a preferred embodiment of the present application, the prior model comprises a probability model of adjacent display of any two commodities, i.e. the context of commodity display. Therefore, the prior model is a probability model. Preferably, by the prior model, input of two commodity categories [A, B] outputs the probability P(A, B) of their adjacent display. For example, input of [Coke, Sprite] outputs the probability P(Coke, Sprite) of adjacent display of Coke and Sprite as 80%, input of [Coke, Red Bull] outputs the probability P(Coke, Red Bull) of adjacent display of Coke and Red Bull as 10%, input of [Coke, Yogurt] outputs the probability P(Coke, Yogurt) of adjacent display of Coke and Yogurt as 1%, and input of [Coke, cosmetic spray] outputs the probability P(Coke, cosmetic spray) of adjacent display of Coke and cosmetic spray as 0. Meanwhile, the prior model can also comprise left and right adjacent display. For example, according to the requirement of in-store display, small, medium and large commodities of a cosmetic cotton must be displayed in sequence, and the probability of small cosmetic cotton right to medium cosmetic cotton is 100%, the probability of small cosmetic cotton right to large cosmetic cotton is 0, the probability of medium cosmetic cotton left to small cosmetic cotton is 100%, the probability of medium cosmetic cotton right to large cosmetic cotton is 100%, the probability of medium cosmetic cotton right to small cosmetic cotton is 0, the probability of medium cosmetic cotton left to large cosmetic cotton is 0, the probability of large cosmetic cotton left to medium cosmetic cotton is 100%, and the probability of large cosmetic cotton left to small cosmetic cotton is 0. For example, according to the requirement of in-store display, multiple different brands and different volumes of Coke must be displayed in a cluster, and the probability of adjacent display of a brand of Coke and another brand of Coke is relatively high, while the probability of adjacent display of Coke and other beverages is relatively low. For example, according to the requirement of in-store display, in order to facilitate customer selection, vegetable salad and salad dressing are displayed side by side, and when input is [vegetable salad, salad dressing], the output is the probability P(vegetable salad, salad dressing) of adjacent display, which is relatively high, and when vegetable salad is input with other commodities, the output is the probability of adjacent display, which is relatively low. The prior model is established according to the context of commodity display, and can be used for commodity recognition, replenishment reminding, error correction of incorrect replenishment, or error correction of arbitrary movement of display commodity sequence in a self-selection supermarket, which are all within the protection scope of the present application.

[0045] In addition, for some shopping malls or convenience stores, strict commodity arrangement modes are formulated according to big data such as geographical location, consumer groups and consumption habits, for example, what kind of commodities are placed on each shelf and each shelf position, the quantity of commodities, and the display mode of commodities are strictly limited. Therefore, according to an embodiment of the present application, the prior model can comprise a preset commodity arrangement mode.

[0046] In step S102, the displayed commodities are recognized by image recognition to obtain a first recognition result.

[0047] The existing commodity recognition algorithm extracts features of a commodity picture through a commodity recognition model, and then compares the extracted features with a database to find the closest features. The model of the commodity recognition algorithm is obtained through machine learning by inputting a large number of commodity pictures, so as to extract features of different commodities and recognize specific commodities according to the extracted features. The features include, but are not limited to, one or more of the color, shape, picture, and size of the commodity. In addition, the first recognition result obtained for each commodity does not necessarily include only one commodity, but can have multiple candidate commodities, which are sorted according to their probabilities or execution degrees, which will be described in detail below.

[0048] According to a preferred embodiment of the present application, step S102 further comprises:

[0049] Taking a picture of the commodity on the shelf;

[0050] Performing image recognition on the commodity picture to generate the first recognition result.

[0051] According to a preferred embodiment of the present application, features are extracted from the commodity picture by the commodity recognition algorithm, and a detection box is generated at the same time. Then, the extracted feature information is compared with the feature information stored in the database in the form of data, and the commodity ID corresponding to the selected feature is the commodity recognition result. The commodity ID includes the type and name of the commodity. Since the comparison of features is based on the similarity of features, the commodity recognition result obtained is not unique, and can be arranged from high to low according to the confidence. According to a preferred embodiment of the present application, as shown in Figure 2 the commodity recognition result and the corresponding confidence are arranged from high to low to form a probability vector as the first recognition result.

[0052] According to another preferred embodiment of the present application, the commodity picture is directly input into a commodity recognition model to obtain the commodity category. The commodity recognition model is obtained through machine learning by inputting a large number of commodity pictures, and the model is trained to directly obtain the type of the commodity. Similarly, the commodity recognition result and the corresponding confidence are arranged from high to low to form a probability vector as the first recognition result.

[0053] However, only through the commodity recognition model, there may be a certain error rate, especially when there are various influencing factors in the image acquisition site environment, the error rate will be high, for example, when there are obstructions, reflections, and strong colored lighting in the field. As Figure 3 shown, three commodities displayed side by side are three different brands of masks. The commodity picture is recognized by the commodity recognition algorithm, preferably three detection boxes are generated, and the commodity feature information is compared with the feature information stored in the database in the form of data to obtain the commodity recognition result pointed to by the arrow in the figure. Figure 3The commodity recognition results shown in the figure are arranged according to the confidence from high to low after feature comparison, and the recognition result with the highest confidence is selected as the commodity recognition result. Among them: the commodity recognition result corresponding to the mask of the first brand is a charger of a certain brand, the commodity recognition result corresponding to the mask of the second brand is a snack of a certain brand, and the commodity recognition result corresponding to the mask of the third brand is the same as itself. Due to the reflection when shooting, the confidence of the correct recognition result is not the highest, so the mask in the first detection box is recognized as a charger, and this recognition result is obviously wrong. For the mask in the second detection box, because the color of the mask packaging is similar to the color of a certain brand of snack in the database, the highest confidence result in the recognition result is also a snack, and this recognition result is obviously wrong. The mask in the third detection box is compared with the feature information in the database after feature extraction, and the highest confidence recognition result output is consistent with the actual commodity.

[0054] In step S103, the first recognition result is corrected according to the prior model to obtain a second recognition result.

[0055] The first recognition result obtained in step S102 is corrected by the prior model obtained in step S101, that is, the preliminary result of image recognition is combined with the context information of the commodity to obtain a corrected recognition result. As in the above Figure 3 embodiment, according to the prior model, the probability of adjacent arrangement of a charger of a certain brand and a snack of a certain brand is very low, that is, P(charger, snack) = 0, so this arrangement combination can be excluded, and other arrangement combinations arranged from high to low in confidence are scored through the traversal method or other algorithms. The scoring refers to finding the arrangement combination with the highest comprehensive confidence, and taking the arrangement combination as the second recognition result. Preferably, the arrangement combination with the highest comprehensive confidence can be found by a set of search algorithms, that is, the final recognition result in the figure.

[0056] As understood by those skilled in the art, the step S101 of obtaining the prior model can be performed in real time or in advance, and the step S101 and the step S102 do not have a certain order.

[0057] According to a preferred embodiment of the present invention, the product identification method 10 further includes: generating a confidence matrix based on the first identification result, wherein the row vectors of the confidence matrix are the first identification results of products sequentially displayed on the same layer, and the column vectors of the confidence matrix are the probability vectors of products displayed at that position; calculating a comprehensive confidence score for each combination sequence in the confidence matrix based on the prior model; and obtaining a second identification result based on the calculation result of the comprehensive confidence score. The second identification result is the combination sequence with the highest comprehensive confidence score and its corresponding product identification result.

[0058] The overall confidence level can be obtained through various algorithms. The following describes several methods for obtaining the overall confidence level.

[0059] Example One

[0060] like Figure 4 As shown, taking a shelf with three displayed products on one floor as an example, the process involves photographing the products on that shelf. These photos are then used to extract features using a product recognition algorithm, simultaneously generating bounding boxes. The extracted features are compared with features stored in a database to obtain a product recognition result including confidence levels. These product recognition results, including confidence levels, are arranged in descending order of confidence to form a probability vector, which serves as the first recognition result. Figure 4 As shown, the first identification results for product 1 are [charger, 60%; mask 1, 35%; night light, 5%], for product 2 they are [beverage, 50%; mask 2, 35%; shampoo, 15%], and for product 3 they are [mask 3, 40%; cotton pad, 35%; notebook, 25%]. The first identification results of all products displayed on the same floor form a confidence matrix.

[0061] In this embodiment, the top three product identification results with the highest confidence are taken as the first identification result. The three possible products with the highest confidence at each product position are connected to the three possible products with the highest confidence at the adjacent product positions, resulting in the relationship network shown in the figure. The overall confidence of each chain is the confidence of the product identification result at that product position multiplied by the probability that the adjacent product positions are connected by a line.

[0062] Each combination sequence in the confidence matrix is ​​scored, that is, the overall confidence is calculated based on the prior model.

[0063] like Figure 4As shown, for each possible combination of connections, multiply the confidence contained in the first identification result by the probability of adjacent display of two commodities (obtained through the prior model), and then add them to obtain the comprehensive confidence of the combination. Take the combination with the highest comprehensive confidence as the second identification result, which includes the updated commodity identification result.

[0064] For example, for the identified commodities 1-3, each commodity has 3 alternative results, and there are 3*3*3=27 combinations, and the comprehensive confidence of each combination is calculated. For example, for the combination of charger-mask 2-laptop, the comprehensive confidence is calculated as follows:

[0065] [60%*P(charger, mask)+35%*P(charger, mask)]+[35%*P(mask, laptop)+25%*P(mask, laptop)]

[0066] After obtaining the comprehensive confidence of each combination, take the combination with the highest comprehensive confidence as the second identification result.

[0067] The above embodiment takes the top three commodity identification results according to the confidence, which can be adjusted according to actual application. The commodity identification method provided by the present application can include technical solutions that take more or fewer commodity identification results according to the confidence, which are within the protection scope of the present application.

[0068] Example Two

[0069] In the above embodiment, the posterior probability is corrected by the traversal method, which has a large amount of calculation. In the present embodiment, the first identification result is corrected by the bundle search method, which will be explained in detail below.

[0070] According to a preferred embodiment of the present application, the current display surface has N display commodities in a layer, and the identification result corresponding to the nth display commodity is Where k∈1,...,K represents the first K identification results, R represents the specific identification result, and S represents the corresponding identification score (i.e. confidence).

[0071] Given the prior model M, the correlation between any two identification results M(R1,R2) can be given. For the identification result of this layer, find a sequence X=x1,...,x N , where x n represents the selection of the x n th identification result of the nth display commodity, and the total single-layer identification confidence C(X) is obtained:

[0072]

[0073] The X* is needed to be found so that the corresponding C(X*) is maximum.

[0074] There are many ways to get this optimal sequence, for example, a greedy algorithm can be used, that is, starting from the first display commodity, for the next display commodity, the result that maximizes the overall confidence of the new sequence is selected each time, that is, assuming x1,...,x i After the selection is made, the selected x i+1 satisfies is maximum.

[0075] The other extreme is to exhaust all possible combinations of X, and then calculate the corresponding confidence one by one, and select the maximum.

[0076] According to a preferred embodiment of the present application, a bundle search algorithm is used to obtain the sequence, while a balance between speed and quality can be obtained. Specifically, the selection is made according to the strategy of the greedy algorithm for each display commodity, but P best sequences are retained each time. Assuming x1,...,x i After the selection is made, there are P sequences corresponding to X 1 ,...,X P , the identification result of the next display commodity is added to each sequence, and the possibility is tried, that is, there will be P*K sequences of length i+1. The confidence of each sequence is calculated, and then only the first P of them are retained.

[0077] Both of the above two embodiments are corrected by the first identification result of the entire display commodity and the prior model. Those skilled in the art can understand that in the case of 1 / 2 layer, 1 / 3 layer and other less than one layer, the identification and verification of the display commodity are also feasible by using the commodity identification method provided by the present application, and the identification result is corrected by the adjacent probability of the display commodity, which is within the protection scope of the present application.

[0078] According to a preferred embodiment of the present application, in the commodity identification method 10, the second identification result is the combination sequence with the highest comprehensive confidence and the corresponding commodity identification result.

[0079] According to a preferred embodiment of the present application, in the commodity identification method 10, the step S103 is realized by a bundle search algorithm or a Viterbi algorithm.

[0080] The present application also provides a computer readable storage medium, comprising computer executable instructions stored thereon, the executable instructions being executed by a processor to implement the commodity identification method 10 as described above.

[0081] The preferred embodiment of the present application provides a commodity identification method, which establishes a prior model of commodity display through historical display data and / or in-store display requirements in a supermarket or convenience store, detects or identifies the displayed commodities through image recognition, and corrects the detection or identification results through the prior model. The preferred embodiment of the present application improves the accuracy of commodity identification and provides a necessary condition for the unmanned of supermarkets / convenience stores.

[0082] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or equivalently replace some technical features thereof. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A product identification method, characterized in that, include: S101: Establish a priori model based on the store's product display habits; S102: Identify the displayed merchandise using image recognition to obtain the first identification result; S103: Correct the first recognition result according to the prior model to obtain the second recognition result. The step S101 further includes: establishing a probability model based on the probability of any two products being displayed adjacently, as the prior model. Step S103 further includes: For products displayed on the same shelf, a confidence matrix is ​​generated based on the first identification result. The row vector of the confidence matrix is ​​the first identification result of the products displayed sequentially on the same shelf, and the column vector of the confidence matrix is ​​the probability vector of the products displayed at that position. Based on the prior model, calculate the overall confidence of the sequence for some or all of the combined sequences in the confidence matrix; The second identification result is obtained based on the calculation result of the comprehensive confidence level.

2. The identification method as described in claim 1, wherein the in-store merchandise display habits include historical display data and / or in-store display requirements.

3. The identification method as described in claim 1, wherein step S102 further comprises: Take photos of the products on the shelves; The product photo is subjected to image recognition to generate the first recognition result.

4. The identification method as described in claim 3, further comprising: Perform target detection on the product photos to generate one or more detection boxes; Feature extraction is performed on the target within the detection box; The features are compared with the database; The first identification result is obtained, which includes the product identification result and the corresponding confidence level.

5. The identification method as described in claim 1, wherein the first identification result is a probability vector, including confidence levels arranged from high to low and their corresponding product identification results.

6. The identification method as described in claim 1, wherein the second identification result is the combined sequence with the highest overall confidence and its corresponding product identification result.

7. The identification method as described in claim 1, wherein step S103 is implemented by a cluster search algorithm or a Viterbi algorithm.

8. A computer-readable storage medium comprising computer-executable instructions stored thereon, the executable instructions, when executed by a processor, implementing the identification method as described in any one of claims 1-7.

Images