Commodity identification method, device and equipment, medium and program product

Abstract

The invention discloses a commodity identification method, device and equipment, a medium and a program product. The commodity identification method comprises the following steps: respectively inputting a commodity image of a to-be-identified commodity into a commodity classification model and a commodity retrieval model, and obtaining a category confidence degree set output by the commodity classification model and a category similarity degree set output by the commodity retrieval model based on a commodity feature library; under the condition that the maximum category confidence coefficient in the category confidence coefficient set is greater than or equal to a confidence coefficient threshold value, taking the commodity category corresponding to the maximum category confidence coefficient as a commodity identification result; and under the condition that the maximum category confidence is smaller than a confidence threshold, determining a commodity identification result according to the category confidence set and the category similarity set. According to the technical scheme of the embodiment of the invention, the commodity identification accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a method, device, equipment, medium and program product for commodity recognition. Background Art

[0002] With the rapid development of artificial intelligence technology, artificial intelligence technology has gradually entered the supermarket field, promoting the development of supermarkets towards digitalization and intelligentization.

[0003] Intelligent digital shelves are the future development trend in the intelligent retail business. Functions such as commodity detection, commodity recognition, out-of-stock detection, and display supervision on the shelves endow the intelligent management requirements of intelligent digital shelves.

[0004] Commodity recognition, as an important part of intelligent shelf management and the basis for technologies such as shelf layout analysis and out-of-stock analysis of shelves, is one of the key technologies for realizing intelligent shelf management. Therefore, how to improve the accuracy of commodity recognition is very important for intelligent shelf management. Summary of the Invention

[0005] The present invention provides a method, device, equipment, medium and program product for commodity recognition to solve...

[0006] According to one aspect of the present invention, there is provided a method for commodity recognition, including:

[0007] Inputting the commodity image of the commodity to be recognized into a commodity classification model and a commodity retrieval model respectively, and obtaining the set of category confidence degrees output by the commodity classification model and the set of category similarities output by the commodity retrieval model based on a commodity feature library;

[0008] The set of category confidence degrees includes the probability that the commodity to be recognized belongs to each commodity category, and the set of category similarities includes the similarity between the commodity to be recognized and each commodity category in the commodity category library;

[0009] In the case where the maximum category confidence degree in the set of category confidence degrees is greater than or equal to a confidence threshold, taking the commodity category corresponding to the maximum category confidence degree as the commodity recognition result;

[0010] In the case where the maximum category confidence degree is less than the confidence threshold, determining the commodity recognition result according to the set of category confidence degrees and the set of category similarities.

[0011] According to another aspect of the present invention, there is provided a commodity recognition device, including:

[0012] A confidence acquisition module, configured to input the product image of the product to be recognized into a product classification model and a product retrieval model respectively, and acquire a set of category confidence degrees output by the product classification model and a set of category similarity degrees output by the product retrieval model based on a product feature library;

[0013] The set of category confidence degrees includes the probabilities that the product to be recognized belongs to each product category, and the set of category similarity degrees includes the similarity degrees between the product to be recognized and each product category in the product category library;

[0014] A first recognition result determination module, configured to, when the maximum category confidence degree in the set of category confidence degrees is greater than or equal to a confidence threshold, use the product category corresponding to the maximum category confidence degree as the product recognition result;

[0015] A second recognition result determination module, configured to, when the maximum category confidence degree is less than the confidence threshold, determine a product recognition result according to the set of category confidence degrees and the set of category similarity degrees.

[0016] According to another aspect of the present invention, there is provided an electronic device, including:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the product recognition method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the product recognition method according to any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, there is provided a computer program product including a computer program, and when the computer program is executed by a processor, it implements the product recognition method according to any embodiment of the present disclosure.

[0022] In the technical solution of the embodiment of the present invention, the product image of the product to be recognized is input into the product classification model and the product retrieval model respectively, and the set of category confidence degrees output by the product classification model and the set of category similarity degrees output by the product retrieval model based on the product feature library are obtained. Furthermore, when the maximum category confidence degree in the set of category confidence degrees is greater than or equal to the confidence degree threshold, the product category corresponding to the maximum category confidence degree is used as the product recognition result. When the maximum category confidence degree is less than the confidence degree threshold, the product recognition result is determined according to the set of category confidence degrees and the set of category similarity degrees. By combining the product classification model and the product retrieval model for product recognition, the accuracy and reliability of product recognition are improved.

[0023] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1a is a flowchart of a product recognition method provided in Embodiment 1 of the present invention;

[0026] Figure 1b is a schematic diagram of data processing of a new classification model provided in Embodiment 1 of the present invention;

[0027] Figure 2a is a flowchart of a product recognition method provided in Embodiment 2 of the present invention;

[0028] Figure 2b is a schematic diagram of the product recognition process provided in Embodiment 2 of the present invention;

[0029] Figure 3 is a schematic diagram of the structure of a product recognition device provided in Embodiment 3 of the present invention;

[0030] Figure 4 is a schematic diagram of the structure of an electronic device for implementing the product recognition method of the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work shall fall within the protection scope of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1a A flowchart of a commodity recognition method is provided for Embodiment 1 of the present invention. This embodiment is applicable to the situation of combining a commodity classification method and a commodity retrieval method for commodity recognition. This method can be executed by a commodity recognition device, which can be implemented in the form of hardware and / or software, and the commodity recognition device can be configured in various general computing devices. As Figure 1a shown, the method includes:

[0035] S110. Input the commodity image of the commodity to be recognized into a commodity classification model and a commodity retrieval model respectively, and obtain the set of category confidence degrees output by the commodity classification model and the set of category similarities output by the commodity retrieval model based on the commodity feature library.

[0036] Among them, the set of category confidence degrees includes the probabilities that the commodity to be recognized belongs to each commodity category, and the set of category similarities includes the similarities between the commodity to be recognized and each commodity category in the commodity category library.

[0037] The commodity classification model is used to classify the commodity to be recognized included in the commodity image and output the probabilities that the commodity to be recognized belongs to each commodity category. The commodity classification model is trained according to existing commodity images and category labels of the commodity images. Exemplarily, the commodity classification model is trained based on deep learning methods according to existing commodity images and category labels of the commodity images.

[0038] The commodity retrieval model includes a feature extraction module and a feature comparison module, which are used to extract features from the commodity image of the commodity to be recognized through the feature extraction module to obtain the feature vector to be recognized, and then compare the feature vector to be recognized with the feature vectors of each commodity category pre-stored in the commodity feature library through the feature comparison module, and output the similarity between the commodity to be recognized and each commodity category in the commodity category library. Among them, the commodity feature library stores the feature vectors corresponding to each commodity category. The commodity retrieval model is trained based on existing commodity images and the category labels of the commodity images.

[0039] In the embodiment of the present invention, after obtaining the commodity image of the commodity to be recognized input by the user, the commodity image is respectively input into the commodity classification model and the commodity retrieval model. The commodity classification model directly classifies the commodity to be recognized and outputs a set of category confidence levels. Among them, the set of category confidence levels includes the probabilities that the commodity to be recognized belongs to each commodity category.

[0040] The commodity retrieval model extracts features from the commodity image, sequentially compares the extracted feature vectors with multiple feature vectors in the commodity feature library, and outputs a set of category similarities. Among them, the set of category similarities includes the similarities between the commodity to be recognized and each commodity category in the commodity category library.

[0041] Optionally, before respectively inputting the commodity image of the commodity to be recognized into the commodity classification model and the commodity retrieval model, it further includes:

[0042] Obtain the commodity images corresponding to each commodity category, and extract features from the commodity images to obtain commodity features;

[0043] Correspondingly store the commodity features and the commodity categories into the commodity feature library.

[0044] In this optional embodiment, specific steps before respectively inputting the commodity image of the commodity to be recognized into the commodity classification model and the commodity retrieval model are provided: obtain the commodity images corresponding to each commodity category, and extract features from the commodity images to obtain commodity features. Then, correspondingly store the commodity features and the commodity categories into the commodity feature library.

[0045] In a specific example, extract features from the commodity image of the drink of brand A to obtain commodity features, and correspondingly store the commodity features and the drink of brand A into the commodity feature library to provide a retrieval basis for the subsequent commodity retrieval model.

[0046] The commodity retrieval model does not need to learn all commodity categories. When adding a new commodity category, as long as the commodity images belonging to the new commodity category are subjected to feature extraction, and the extracted commodity features are correspondingly stored in the commodity feature library together with the commodity category, they can be correctly retrieved subsequently, which can improve the recognition accuracy of the new commodity category.

[0047] S120. When the maximum class confidence in the class confidence set is greater than or equal to the confidence threshold, use the product category corresponding to the maximum class confidence as the product recognition result.

[0048] By effectively learning the features and differences between different products, the product classification model can accurately identify the product type to which the learned products belong in most cases. However, if it encounters new products that have not been learned, identification errors may occur.

[0049] In the embodiment of the present invention, after obtaining the class confidence set output by the product classification model, first obtain the maximum class confidence in the set, and compare the maximum class confidence with the pre-set confidence threshold. If the maximum class confidence is greater than or equal to the confidence threshold, it means that the output result of the current product classification model is highly reliable, and directly use the product category corresponding to the maximum class confidence as the product recognition result.

[0050] S130. When the maximum class confidence is less than the confidence threshold, determine the product recognition result according to the class confidence set and the class similarity set.

[0051] In the embodiment of the present invention, if the maximum class confidence is less than the confidence threshold, determine the product recognition result according to the class confidence set and the class similarity set. Specifically, the maximum value can be obtained in the class confidence set, that is, P CTop-1 . And obtain the maximum value in the class similarity set, that is, P RTop-1 . Then compare the two, and select the product category corresponding to the maximum value as the product recognition result.

[0052] It is also possible to take the first n values in the class confidence set in descending order, that is, P CTop-n . And take the first n values in the class similarity set in descending order, that is, P RTop-n . Finally, if the product category belongs to both P CTop-n and P RTop-n , then perform a weighted sum of the class confidence and class similarity corresponding to the product category to obtain a comprehensive confidence, where the weight of the class confidence is the product classification fusion coefficient, and the weight of the class similarity is the product retrieval fusion coefficient; if the product category only belongs to P CTop-n multiply the class confidence corresponding to the product category by the product classification fusion coefficient to obtain a comprehensive confidence; if the product category only belongs to P RTop-n , then multiply the class similarity corresponding to the product category by the product retrieval fusion coefficient to obtain a comprehensive confidence. Finally, use the product category with the maximum comprehensive confidence as the product recognition result.

[0053] The category confidence set and the category similarity set can also be jointly input into a pre-trained new classification model, and the product category with the highest confidence output by the new classification model is determined as the product recognition result. Among them, the new classification model is trained based on the category confidence set, the category similarity set corresponding to the existing product images, and the category labels of the product images, using a deep learning algorithm.

[0054] The number of elements included in the category confidence set and the category similarity set associated with the product image of the product to be recognized is the same. All the elements in the category confidence set are combined into a category confidence vector. Similarly, all the elements in the category similarity set are combined into a category similarity vector. Then, the category confidence vector and the category similarity vector are input into the new classification model. The processing method of the new classification model is specifically as Figure 1b shown: First, the category confidence vector and the category similarity vector are respectively dimensionally reduced to the same dimension; further, through N1 layers of networks, feature learning is performed on the dimensionally reduced category confidence vector and the dimensionally reduced category similarity vector, respectively obtaining the intermediate layer feature vector associated with the category confidence vector and the intermediate layer feature vector associated with the category similarity vector; further, the intermediate layer vector associated with the category confidence vector and the intermediate layer vector associated with the category similarity vector are added together, and feature learning is performed on the added vector through N2 layers of networks to obtain a classification feature vector; further, through the classification layer, the classification feature vector is processed to output the new category confidence of each product category; finally, based on the new category confidence output by the new classification model, the product category corresponding to the maximum category confidence in the new category confidence is used as the product recognition result.

[0055] In the technical solution of this embodiment of the present invention, the product image of the product to be recognized is respectively input into the product classification model and the product retrieval model, and the category confidence set output by the product classification model and the category similarity set output by the product retrieval model based on the product feature library are obtained. Then, in the case where the maximum category confidence in the category confidence set is greater than or equal to the confidence threshold, the product category corresponding to the maximum category confidence is used as the product recognition result. In the case where the maximum category confidence is less than the confidence threshold, the product recognition result is determined according to the category confidence set and the category similarity set. By combining the product classification model and the product retrieval model for product recognition, the accuracy and reliability of product recognition are improved.

[0056] Embodiment 2

[0057] Figure 2a It is a flowchart of a product recognition method provided by Embodiment 2 of the present invention. This embodiment is further refined on the basis of the above embodiment, and provides specific steps for determining the product recognition result according to the category confidence set and the category similarity set. AsFigure 2a As shown in the figure, the method includes:

[0058] S210. Input the product image of the product to be recognized into the product classification model and the product retrieval model respectively, and obtain the set of class confidence degrees output by the product classification model and the set of class similarity degrees output by the product retrieval model based on the product feature library.

[0059] The set of class confidence degrees includes the probabilities that the product to be recognized belongs to each product class, and the set of class similarity degrees includes the similarity degrees between the product to be recognized and each product class in the product class library.

[0060] S220. When the maximum class confidence degree in the set of class confidence degrees is greater than or equal to the confidence degree threshold, use the product class corresponding to the maximum class confidence degree as the product recognition result.

[0061] S230. When the maximum class confidence degree is less than the confidence degree threshold, sort the class confidence degrees in the set of class confidence degrees from large to small, and sequentially extract the first quantity of class confidence degrees according to the sorting result to form the first confidence degree set.

[0062] In the case of a large number of product classes, there are a large number of values in the set of class confidence degrees and the set of class similarity degrees, and the calculation amount is very large during the fusion process. In the set of class confidence degrees and the set of class similarity degrees, the correct product recognition result often exists in a part of the class confidence degrees with larger class confidence degrees or a part of the class similarity degrees with larger class similarity degrees.

[0063] In the embodiments of the present invention, the specific product recognition process is as Figure 2b shown. In order to reduce the fusion calculation amount, sort the class confidence degrees in the set of class confidence degrees from large to small, and sequentially extract the first quantity X of class confidence degrees, that is, P CTop-X , to form the first confidence degree set.

[0064] In a specific example, the set of class confidence degrees output by the product classification model includes 100 class confidence degrees. Sort these 100 class confidence degrees from large to small, and sequentially extract 20 class confidence degrees according to the sorting result to form the first confidence degree set.

[0065] S240. Sort the class similarity degrees in the set of class similarity degrees from large to small, and sequentially extract the first quantity of class similarity degrees according to the sorting result to form the first similarity degree set.

[0066] In the embodiments of the present invention, in order to reduce the fusion calculation amount, sort the class similarity degrees in the set of class similarity degrees from large to small, and sequentially extract the first quantity X of class similarity degrees, that is, P RTop-X, form the first similarity set.

[0067] In a specific example, the category similarity set output by the commodity retrieval model includes 100 category similarities. These 100 category similarities are sorted from largest to smallest, and 20 category similarities are extracted in sequence according to the sorting result to form the first similarity set.

[0068] By separately taking the top first quantity of values in the category confidence set and the category similarity set in descending order, the subsequent calculation amount can be reduced and the commodity recognition efficiency can be improved.

[0069] S250. Determine the commodity recognition result according to the first confidence set and the first similarity set.

[0070] In the embodiment of the present invention, the commodity recognition result is determined according to the first confidence set and the first similarity set. Specifically, if the commodity category belongs to both the first confidence set and the first similarity set, the category confidence and the category similarity corresponding to the commodity category are weighted and summed to obtain a comprehensive confidence, where the weight of the category confidence is the commodity classification fusion coefficient, and the weight of the category similarity is the commodity retrieval fusion coefficient; if the commodity category only belongs to the first confidence set, the category confidence corresponding to the commodity category is multiplied by the commodity classification fusion coefficient to obtain a comprehensive confidence; if the commodity category only belongs to the first similarity set, the category similarity corresponding to the commodity category is multiplied by the commodity retrieval fusion coefficient to obtain a comprehensive confidence. Finally, the commodity category with the largest comprehensive confidence is used as the commodity recognition result.

[0071] In addition, before calculating the comprehensive confidence, a normalization operation can be performed on the category confidence in the first confidence set and the category similarity in the first similarity set. Then, the comprehensive confidence is calculated based on the normalized category confidence and category similarity.

[0072] Optionally, determining the commodity recognition result according to the first confidence set and the first similarity set includes:

[0073] Perform a normalization operation on the category confidence in the first confidence set and the category similarity in the first similarity set respectively;

[0074] In the normalization result of the first confidence set, extract the second quantity of category confidences to form a second confidence set; the first quantity is greater than the second quantity;

[0075] In the normalization result of the first similarity set, extract the second quantity of category similarities to form a second similarity set;

[0076] Fuse the second confidence set and the second similarity set to obtain the comprehensive confidence of the product category;

[0077] Based on the comprehensive confidence of the product category, determine the product recognition result.

[0078] In this optional embodiment, a specific method for determining the product recognition result according to the first confidence set and the first similarity set is provided: Since the classification confidence and the classification similarity are the output results of different models, the classification confidence and the classification similarity are in different distribution spaces. In order to fuse the output results of the product classification model and the product retrieval model and improve the product recognition accuracy. Normalize the category confidence in the first confidence set and the category similarity in the first similarity set respectively, for example, perform a softmax operation. Furthermore, in order to further reduce the computational complexity, in the normalized result of the first confidence set , extract the category confidence of the second quantity n to form the second confidence set where the first quantity X is greater than the second quantity n. Similarly, in the normalized result of the first similarity set , extract the category similarity of the second quantity to form the second similarity set

[0079] Further, fuse the second confidence set and the second similarity set to obtain the comprehensive confidence of the product category. Based on the comprehensive confidence of the product category, determine the product recognition result. For example, if the product category belongs to both the second confidence set and the second similarity set at the same time, then perform a weighted sum of the category confidence (the normalized category confidence) and the category similarity (the normalized category similarity) corresponding to the product category, where the weight of the category confidence is the product classification fusion coefficient and the weight of the category similarity is the product retrieval fusion coefficient; if the product category only belongs to the second confidence set, then multiply the category confidence (the normalized category confidence) corresponding to the product category by the product classification fusion coefficient to obtain the comprehensive confidence; if the product category only belongs to the second similarity set, then multiply the category similarity (the normalized category similarity) corresponding to the product category by the product retrieval fusion coefficient to obtain the comprehensive confidence. Finally, take the product category with the maximum comprehensive confidence as the product recognition result.

[0080] In a specific example, the total number of product categories is 100, X in Top-X is 20, and n in Top-n is 5. Among them, {"a": b} means that the probability that the commodity to be recognized belongs to commodity category a is b.

[0082] By normalizing the category confidence degrees in the first confidence set and the category similarity degrees in the first similarity set, the problem that the data in the two sets are in different distribution spaces can be solved, and the accuracy of commodity recognition can be improved. In addition, after performing the normalization operation, a part of the data is further screened in the first confidence set and the first similarity set, which can further reduce the computational amount of commodity recognition and improve the efficiency of commodity recognition.

[0083] Optionally, the second confidence set and the second similarity set are fused to obtain the comprehensive confidence degree of the commodity category, including:

[0084] Take the union of the commodity categories to which each category confidence degree in the second confidence set belongs and the commodity categories to which each category similarity degree in the second similarity set belongs to obtain a commodity category set;

[0085] Extract a commodity category from the commodity category set in turn as the current category;

[0086] Based on the target category confidence degree and the target category similarity degree associated with the current category, according to the commodity classification fusion coefficient and the commodity retrieval fusion coefficient, the target category confidence degree and the target category similarity degree are fused to obtain the comprehensive confidence degree of the current category, and return to perform the operation of extracting a commodity category from the commodity category set in turn as the current category until the traversal of the commodity categories in the commodity category set is completed.

[0087] In this optional embodiment, a specific method for fusing the second confidence set and the second similarity set to obtain the comprehensive confidence of the commodity category is provided: First, take the union of the commodity categories to which each category confidence in the second confidence set belongs and the commodity categories to which each category similarity in the second similarity set belongs to obtain a commodity category set. Then, sequentially extract a commodity category from the commodity category set as the current category. Based on the target category confidence and the target category similarity associated with the current category, according to the commodity classification fusion coefficient and the commodity retrieval fusion coefficient, fuse the target category confidence and the target category similarity to obtain the comprehensive confidence of the current category. Specifically, if the commodity category belongs to both the second confidence set and the second similarity set, then perform a weighted sum of the category confidence and the category similarity corresponding to the commodity category to obtain the comprehensive confidence, where the weight of the category confidence is the commodity classification fusion coefficient and the weight of the category similarity is the commodity retrieval fusion coefficient; if the commodity category only belongs to the second confidence set, then multiply the category confidence corresponding to the commodity category by the commodity classification fusion coefficient to obtain the comprehensive confidence; if the commodity category only belongs to the second similarity set, then multiply the category similarity corresponding to the commodity category by the commodity retrieval fusion coefficient to obtain the comprehensive confidence. The specific formula for the comprehensive confidence is as follows:

[0088]

[0089] where P fuse,i is the comprehensive confidence of the i-th commodity category, α is the commodity classification fusion coefficient, β is the commodity retrieval fusion coefficient, is the category confidence of the i-th commodity category, is the category similarity of the i-th commodity category.

[0090] Furthermore, return to perform the operation of sequentially extracting a commodity category from the commodity category set as the current category until the traversal of all commodity categories in the commodity category set is completed, and obtain the comprehensive confidence corresponding to all commodity categories in the commodity category set.

[0091] In a specific example, α = 0.5, β = 0.5, then the comprehensive confidences of each commodity category are as follows: P fuse = [{"1": 0.655}, {"8": 0.155}, {"29": 0.004}, {"30": 0.04}, {"50": 0.035}, {"45": 0.03}, {"80": 0.025}, {"88": 0.02}].

[0092] Based on the commodity classification fusion coefficient and the commodity retrieval fusion coefficient, fusing the category confidence and the category similarity can, when the reliability of the result output by the commodity classification model is not high, further combine the output result of the commodity retrieval model, and can improve the recognition accuracy of new commodities while ensuring the recognition accuracy of the learned commodity categories.

[0093] Optionally, before fusing the target category confidence and the target category similarity according to the commodity classification fusion coefficient and the commodity retrieval fusion coefficient, it further includes:

[0094] Determine the commodity classification fusion coefficient and the commodity retrieval fusion coefficient according to the target category confidence associated with the current category;

[0095] Among them, the target category confidence is positively correlated with the commodity classification fusion coefficient, and the target category confidence is negatively correlated with the commodity retrieval fusion coefficient.

[0096] In this optional embodiment, a specific solution before fusing the target category confidence and the target category similarity according to the commodity classification fusion coefficient and the commodity retrieval fusion coefficient is provided: determine the commodity classification fusion coefficient and the commodity retrieval fusion coefficient according to the target category confidence associated with the current category. Among them, the target category confidence is positively correlated with the commodity classification fusion coefficient, and the target category confidence is negatively correlated with the commodity retrieval fusion coefficient. Determining the commodity classification fusion coefficient and the commodity retrieval fusion coefficient according to the target category confidence associated with the current category, the greater the target category confidence, the higher the commodity classification fusion coefficient, which can flexibly adjust the fusion weight and improve the accuracy of commodity recognition.

[0097] In a specific example, assuming that the category confidence threshold is 0.8, and the target category confidence associated with the current category is between 0.7 and 0.8, then the commodity classification fusion coefficient is determined to be 0.7, and the commodity retrieval fusion coefficient is 0.3; if the target category confidence associated with the current category is between 0.5 and 0.7, then the commodity classification fusion coefficient is determined to be 0.6, and the commodity retrieval fusion coefficient is 0.4.

[0098] The technical solution of the embodiment of the present invention, when the maximum category confidence is unreliable, by respectively taking the first number of values in descending order from the category confidence set and the category similarity set, can reduce the subsequent calculation amount and improve the commodity recognition efficiency, and after normalizing the first number of values in the category confidence set and the category similarity set and then performing the fusion operation, can convert the category confidence and the category similarity to the same feature space before performing the fusion, further improving the commodity recognition accuracy.

[0099] Embodiment III

[0100] Figure 3 This is a schematic structural diagram of a commodity recognition device provided in Embodiment 3 of the present invention. As Figure 3 shown, the device includes:

[0101] A confidence level acquisition module 310, configured to input the commodity image of the commodity to be recognized into a commodity classification model and a commodity retrieval model respectively, and acquire a set of category confidence levels output by the commodity classification model and a set of category similarities output by the commodity retrieval model based on a commodity feature library;

[0102] The set of category confidence levels includes the probabilities that the commodity to be recognized belongs to each commodity category, and the set of category similarities includes the similarities between the commodity to be recognized and each commodity category in the commodity category library;

[0103] A first recognition result determination module 320, configured to, when the maximum category confidence level in the set of category confidence levels is greater than or equal to a confidence level threshold, use the commodity category corresponding to the maximum category confidence level as the commodity recognition result;

[0104] A second recognition result determination module 330, configured to, when the maximum category confidence level is less than the confidence level threshold, determine the commodity recognition result according to the set of category confidence levels and the set of category similarities.

[0105] The technical solution of the embodiment of the present invention inputs the commodity image of the commodity to be recognized into a commodity classification model and a commodity retrieval model respectively, and acquires a set of category confidence levels output by the commodity classification model and a set of category similarities output by the commodity retrieval model based on a commodity feature library. Furthermore, when the maximum category confidence level in the set of category confidence levels is greater than or equal to a confidence level threshold, the commodity category corresponding to the maximum category confidence level is used as the commodity recognition result. When the maximum category confidence level is less than the confidence level threshold, the commodity recognition result is determined according to the set of category confidence levels and the set of category similarities. By combining the commodity classification model and the commodity retrieval model for commodity recognition, the accuracy and reliability of commodity recognition are improved.

[0106] Optionally, the second recognition result determination module 330 includes:

[0107] A first confidence level set determination unit, configured to sort the category confidence levels in the set of category confidence levels from large to small, and sequentially extract a first number of category confidence levels according to the sorting result to form a first confidence level set;

[0108] A first similarity set determination unit, configured to sort the category similarities in the set of category similarities from large to small, and sequentially extract a first number of category similarities according to the sorting result to form a first similarity set;

[0109] The second recognition result determination unit is configured to determine the product recognition result according to the first confidence set and the first similarity set.

[0110] Optionally, the second recognition result determination unit includes:

[0111] The normalization subunit is configured to perform normalization operations on the category confidences in the first confidence set and the category similarities in the first similarity set respectively;

[0112] The second confidence set determination subunit is configured to extract the second number of category confidences from the normalization result of the first confidence set to form the second confidence set; the first number is greater than the second number;

[0113] The second similarity set determination subunit is configured to extract the second number of category similarities from the normalization result of the first similarity set to form the second similarity set;

[0114] The comprehensive confidence determination subunit is configured to fuse the second confidence set and the second similarity set to obtain the comprehensive confidence of the product category;

[0115] The second recognition result determination subunit is configured to determine the product recognition result based on the comprehensive confidence of the product category.

[0116] Optionally, the second recognition result determination subunit is specifically configured to:

[0117] Take the union of the product categories to which each category confidence in the second confidence set belongs and the product categories to which each category similarity in the second similarity set belongs to obtain the product category set;

[0118] Successively extract a product category from the product category set as the current category;

[0119] Based on the target category confidence and target category similarity associated with the current category, fuse the target category confidence and target category similarity according to the product classification fusion coefficient and the product retrieval fusion coefficient to obtain the comprehensive confidence of the current category, and return to perform the operation of successively extracting a product category from the product category set as the current category until the traversal of the product categories in the product category set is completed.

[0120] Optionally, the second recognition result determination subunit is further specifically configured to:

[0121] Before fusing the target category confidence and target category similarity according to the product classification fusion coefficient and the product retrieval fusion coefficient, determine the product classification fusion coefficient and the product retrieval fusion coefficient according to the target category confidence associated with the current category;

[0122] Among them, the confidence of the target category is positively correlated with the commodity classification fusion coefficient, and the confidence of the target category is negatively correlated with the commodity retrieval fusion coefficient.

[0123] Optionally, the commodity recognition device further includes:

[0124] A commodity feature extraction module, configured to obtain a commodity image corresponding to each commodity category and extract features from the commodity image to obtain commodity features before inputting the commodity image of the commodity to be recognized into the commodity classification model and the commodity retrieval model respectively;

[0125] A commodity feature storage module, configured to store the commodity features and the commodity categories in correspondence in a commodity feature library.

[0126] The commodity recognition device provided by the embodiments of the present invention can execute the commodity recognition method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0127] In the technical solution of the present invention, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure, and application of relevant data, etc., all comply with relevant laws, regulations, and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0128] Embodiment 4

[0129] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0130] Figure 4 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, an applicator, a blade applicator, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described herein and / or claimed.

[0131] As Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0132] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0133] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the product recognition method.

[0134] In some embodiments, the product recognition method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the product recognition method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the product recognition method by any other appropriate means (e.g., by means of firmware).

[0135] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0136] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0137] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data applicator), or a computing system including middleware components (e.g., an application applicator), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0140] A computing system can include a client and an applicator. The client and the applicator are generally far from each other and usually interact through a communication network. The relationship between the client and the applicator is created by computer programs running on respective computers and having a client-applicator relationship with each other. The applicator can be a cloud applicator, also known as a cloud computing applicator or a cloud host, which is a host product in a cloud computing application system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS applications.

[0141] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0142] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A commodity recognition method, characterized in that, Including: Input the product images of the product to be recognized into the product classification model and the product retrieval model respectively, and obtain the set of category confidence levels output by the product classification model and the set of category similarities output by the product retrieval model based on the product feature library; The set of category confidence levels includes the probabilities that the product to be recognized belongs to each product category, and the set of category similarities includes the similarities between the product to be recognized and each product category in the product category library; When the maximum category confidence level in the set of category confidence levels is greater than or equal to the confidence level threshold, use the product category corresponding to the maximum category confidence level as the product recognition result; When the maximum category confidence level is less than the confidence level threshold, determine the product recognition result according to the set of category confidence levels and the set of category similarities.

2. The method according to claim 1, characterized in that, Determining the product recognition result according to the set of category confidence levels and the set of category similarities includes: Sort the category confidence levels in the set of category confidence levels from largest to smallest, and sequentially extract the first number of category confidence levels according to the sorting result to form a first confidence level set; Sort the category similarities in the set of category similarities from largest to smallest, and sequentially extract the first number of category similarities according to the sorting result to form a first similarity set; Determine the product recognition result according to the first confidence level set and the first similarity set.

3. The method according to claim 2, wherein Determining the product recognition result according to the first confidence level set and the first similarity set includes: Perform normalization operations on the category confidence levels in the first confidence level set and the category similarities in the first similarity set respectively; In the normalized result of the first confidence level set, extract the second number of category confidence levels to form a second confidence level set; the first number is greater than the second number; In the normalized result of the first similarity set, extract the second number of category similarities to form a second similarity set; Fuse the second confidence level set and the second similarity set to obtain the comprehensive confidence level of the product category; Determine the product recognition result based on the comprehensive confidence level of the product category.

4. The method according to claim 3, wherein Fusing the second confidence level set and the second similarity set to obtain the comprehensive confidence level of the product category includes: Take the union of the product categories to which each category confidence level in the second confidence level set belongs and the product categories to which each category similarity in the second similarity set belongs to obtain a product category set; Sequentially extract one product category from the product category set as the current category; Based on the target category confidence level and the target category similarity associated with the current category, fuse the target category confidence level and the target category similarity according to the product classification fusion coefficient and the product retrieval fusion coefficient to obtain the comprehensive confidence level of the current category, and return to perform the operation of sequentially extracting one product category from the product category set as the current category until the traversal of the product categories in the product category set is completed.

5. The method according to claim 4, characterized in that, Before fusing the target category confidence level and the target category similarity according to the product classification fusion coefficient and the product retrieval fusion coefficient, it further includes: Determine the commodity classification fusion coefficient and the commodity retrieval fusion coefficient according to the confidence of the target category associated with the current category; Among them, the confidence of the target category is positively correlated with the commodity classification fusion coefficient, and the confidence of the target category is negatively correlated with the commodity retrieval fusion coefficient.

6. The method according to claim 1, wherein Before inputting the commodity image of the commodity to be recognized into the commodity classification model and the commodity retrieval model respectively, it further includes: Obtain the commodity image corresponding to each commodity category, and perform feature extraction on the commodity image to obtain commodity features; Correspondingly store the commodity features and the commodity category in the commodity feature library.

7. A commodity identification device, characterized in that, It includes: A confidence acquisition module, configured to input the commodity image of the commodity to be recognized into the commodity classification model and the commodity retrieval model respectively, and obtain the set of category confidences output by the commodity classification model and the set of category similarities output by the commodity retrieval model based on the commodity feature library; The set of category confidences includes the probability that the commodity to be recognized belongs to each commodity category, and the set of category similarities includes the similarity between the commodity to be recognized and each commodity category in the commodity category library; A first recognition result determination module, configured to, when the maximum category confidence in the set of category confidences is greater than or equal to the confidence threshold, use the commodity category corresponding to the maximum category confidence as the commodity recognition result; A second recognition result determination module, configured to, when the maximum category confidence is less than the confidence threshold, determine the commodity recognition result according to the set of category confidences and the set of category similarities.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the commodity recognition method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the processor to implement the commodity recognition method according to any one of claims 1-6 when executed.

10. A computer program product, characterized in that, The computer program product includes a computer program, and the computer program implements the commodity recognition method according to any one of claims 1-6 when executed by the processor.

Images