Commodity display identification method and system for retail terminal
Through graphic and text recognition and object detection technology, combined with YOLOv5 and image classification model, the problems of low recognition accuracy and poor adaptability in the existing technology are solved, efficient and accurate retail product recognition is achieved, and maintenance costs are reduced.
Patent Information
- Application Number
- CN202411921116.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-25
- Publication Date
- 2025-04-18
AI Technical Summary
The existing retail product identification methods face diverse product packaging, lighting changes and complex scenarios, have low recognition accuracy, poor adaptability, and high maintenance costs.
The graphic and text recognition and object detection technology are used to manually annotate and preprocess image data, and the object detection model is trained in combination with the YOLOv5 algorithm, and the image classification model is used to classify products. Finally, the feature vector is compared with the Milvus vector library to obtain classification results.
It improves the accuracy and efficiency of product display recognition at retail terminals, reduces the need for manual intervention, has strong robustness and generalization capabilities, and adapts to complex scenarios and similar product categories.
Smart Images
Figure CN119850916A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer application technology, and in particular to a method and system for identifying commodity displays at retail terminals. Background Art
[0002] In daily life, supermarkets sell a wide variety of retail goods, and a huge number of goods are sold every day. The display of different goods at retail terminals has an important impact on the promotion of brand image and consumer purchasing behavior. Marketing decision makers need to quickly and accurately obtain the display status of goods on the shelves of retail terminal stores. However, in actual operation, due to the variety of goods, complex packaging forms, and the influence of factors such as lighting, shooting angles, and occlusion, existing retail product recognition methods face many challenges. The main challenges include a large number of categories of product images, high similarity of outer packaging, a small number of samples per category, and low recognition accuracy.
[0003] At present, most supermarket product display identification methods rely on image recognition technology, which processes pictures of product packaging to identify product names, prices, barcodes and other information. These methods usually rely on predefined rules and templates, or require manual annotation of a large amount of training data. However, with the continuous increase in product types and the diversification of packaging designs, rule- and template-based methods have gradually become inadequate and prone to misidentification and missed identification. At the same time, maintaining these rules and templates also requires a lot of manpower and time costs. Summary of the invention
[0004] The present disclosure aims to solve at least one of the technical problems existing in the prior art, and proposes a method and system for identifying commodity displays at retail terminals.
[0005] In a first aspect, the present disclosure provides a method for identifying commodity displays at a retail terminal, comprising:
[0006] S1, collect image data of commodity displays at retail terminals, and then manually annotate and preprocess the data to obtain preprocessed data;
[0007] S2, based on the image position information in the preprocessed data, uses the YOLOv5 algorithm to train an object detection model to detect the location of the product and price tag in the image;
[0008] S3, based on the small image output by the object detection module, train an image classification model to classify the detected product areas;
[0009] S4, the vector obtained after the product image passes through the detection and image classification model is compared with the Milvus vector library to obtain the final classification result.
[0010] Preferably, the S1 specifically includes:
[0011] S11, collect image data of retail terminal commodity displays of different brands, different packaging forms, different shooting angles, and different lighting conditions;
[0012] S12, preprocessing the image data, including adjusting the resolution of the image data, cutting out the background noise area, and normalizing the image pixels.
[0013] Preferably, after S1 and before S2, the method further includes:
[0014] For the collected image data, the following data augmentation strategies are adopted:
[0015] (1) Randomly rotate the image by a certain angle to simulate different shooting angles;
[0016] (2) Randomly scale images and crop unnecessary areas to enhance the model’s adaptability to different sizes and scenes;
[0017] (3) Randomly adjust the brightness, contrast, and saturation of the image to adapt to different lighting conditions;
[0018] (4) Randomly add noise to increase data diversity and prevent model overfitting.
[0019] Preferably, the S2 specifically includes:
[0020] During the training process, a folding head mechanism is introduced to address the problem of detecting small objects with high resolution.
[0021] Preferably, the S3 specifically includes:
[0022] S31, transforms the classification problem into a metric learning problem, and trains the feature vector representation of each product category through the optimized cosine Softmax loss function, so that the model can distinguish product packaging with extremely high similarity;
[0023] S32, introduces category-level additive margin into the loss function, so that the image classification model can enhance the distinction between different categories by adjusting the interval boundary of each category when performing category classification, thereby improving the model's recognition ability for long-tail and similar categories.
[0024] Preferably, the loss function of the image classification model is as follows:
[0025]
[0026] Among them, L CMS is the loss function used to optimize the model, nNa is the number of categories in the negative sample set, N a is a negative sample set, i.e., a sample set of different categories from the current sample, c is the category to which the current sample belongs, S c is the sample set belonging to the category, x i is sample i, is the category y of sample i i The eigenvector of c is the additive interval of category c, used to adjust the boundaries between categories, is the perturbation vector of category c, used to adjust the feature vector, The norm of the perturbation vector does not exceed the additive interval ε c , s is the scaling factor, which is used to control the influence of the interval, nP a is the positive sample set P a The number of categories in P a is the positive sample set, that is, the sample set of the same category as the current sample, cosθ j is the cosine similarity between the feature vector of category j and the feature vector of sample i.
[0027] Preferably, the S4 specifically includes:
[0028] When a new product image is input, the target detection model is used to identify the product location in the image, and the classification model is used to generate a feature vector for the product. Then, the feature vector is compared with the pre-stored vectors in the Milvus vector library for similarity to obtain the final classification result.
[0029] The present invention also provides a system for identifying commodity displays at retail terminals, which can be used to implement the above-mentioned method for identifying commodity displays at retail terminals, and the system includes:
[0030] A data acquisition module is configured to collect image data of commodity displays at retail terminals, and then manually annotate and preprocess the data to obtain preprocessed data;
[0031] The object detection model training module is configured to train an object detection model using the YOLOv5 algorithm based on the image position information in the preprocessed data to detect the location of the product and price tag in the image;
[0032] The image classification model training module is configured to train an image classification model based on the small image output by the object detection module to classify the detected product areas;
[0033] The output module is configured to compare the vector obtained after the product image passes through the detection and image classification model with the Milvus vector library to obtain the final classification result.
[0034] The present invention uses image and text recognition and target detection technology to quickly and accurately identify the placement of goods on the shelves of retail terminal stores, and converts it into data information that can be analyzed, thereby supporting corresponding marketing decisions. The method combines deep learning and computer vision technology to automatically extract key location information from product images, and then identify them separately without relying on complex rules and templates, and can adapt to changes in various product packaging. The method has strong robustness and generalization capabilities, and can significantly improve the accuracy and efficiency of supermarket product display recognition, reducing the need for manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A flowchart of a method for identifying commodity displays at a retail terminal provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0036] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0037] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure are not any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" are not limited to quantity, but are at least one. "Including" or "including" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used for relative positional relationships. When the absolute position of the described target changes, the relative positional relationship may also change accordingly.
[0038] In each of the accompanying drawings, the same elements are represented by similar reference numerals. For the sake of clarity, the various parts in the accompanying drawings are not all drawn to scale. In addition, some well-known parts may not be shown in the drawings.
[0039] Many specific details of the present disclosure are described below, such as component structures, materials, dimensions, processing techniques, and technologies, in order to more clearly understand the present disclosure. However, as those skilled in the art will appreciate, the present disclosure may be implemented without following these specific details.
[0040] like Figure 1As shown, an embodiment of the present invention provides a method for identifying commodity displays at a retail terminal, comprising:
[0041] S1, collect image data of commodity displays at retail terminals, and then manually annotate and preprocess the data to obtain preprocessed data;
[0042] S2, based on the image position information in the preprocessed data, uses the YOLOv5 algorithm to train an object detection model to detect the location of the product and price tag in the image;
[0043] S3, based on the small image output by the object detection module, train an image classification model to classify the detected product areas;
[0044] S4, the vector obtained after the product image passes through the detection and image classification model is compared with the Milvus vector library to obtain the final classification result.
[0045] By combining image and text recognition and target detection technology, the placement of goods on the shelves can be quickly and accurately identified, and the information can be converted into data for analysis to support product marketing decisions. The system includes target detection and image classification modules, wherein the target detection module uses the YOLOv5 algorithm to locate the position of cigarette boxes and price tags, while the image classification module uses resnet for classification. In view of the problems of small target detection, poor data quality, high similarity and long-tail distribution in the existing schemes, the present invention proposes a series of optimization schemes. A folding head mechanism is introduced in target detection to support higher resolution input and reduce information loss. In the classification algorithm, the classification problem is converted into a metric learning problem, and the recognition accuracy of similar products is improved by using the cosine Softmax loss function with category-level additive intervals. The technical solution of the present invention provides an efficient and accurate solution for display recognition at commodity retail terminals, which helps to improve the efficiency of brand image promotion and consumer purchasing behavior analysis.
[0046] This method aims to solve the problems of poor data set quality, low recognition accuracy, poor adaptability, and high maintenance cost in the process of supermarket commodity recognition in the prior art. Through the present invention, only a part of representative commodity images need to be labeled, and the training data set is expanded by using data augmentation technology. Finally, by combining the features of commodity images with text information, a retail commodity display recognition model is trained using YOLO and Resnet networks, which can efficiently and accurately identify supermarket commodities and extract their key information. This method is suitable for commodity recognition in a variety of complex scenarios and has strong robustness and versatility.
[0047] It should be noted that the technical solution of the present disclosure does not limit the sequence of the above steps, that is, the execution order of the steps can be arranged arbitrarily.
[0048] In a specific implementation scenario, the steps are as follows:
[0049] Step S1: Collect image data of commodity displays at retail terminals, and then manually annotate and preprocess the data to obtain preprocessed data.
[0050] Collect a large number of representative product display images, including different brands, packaging forms, and various shooting angles and lighting conditions. Preprocessing includes adjusting image resolution, cropping background noise areas, normalizing image pixel values, etc., in order to optimize subsequent model training.
[0051] The preferred solution is to perform data augmentation on the preprocessed data to improve the generalization ability of the model. For the collected image data, the following data augmentation strategies are adopted:
[0052] 1) Random rotation: Randomly rotate the image by a certain angle to simulate different shooting angles.
[0053] 2) Scaling and Cropping: Randomly scale images and crop unnecessary areas to enhance the model’s adaptability to different sizes and scenes.
[0054] 3) Color adjustment: Randomly adjust the brightness, contrast, and saturation of the image to adapt to different lighting conditions.
[0055] 4) Adding noise: Noise is randomly added to increase data diversity and prevent model overfitting.
[0056] Step S2: After the data preprocessing is completed, a target detection model is trained using the YOLOv5 algorithm based on the image position information in the preprocessed data to detect the position of the goods and price tags in the image, that is, based on the position information in the large image, a target detection model is trained using the YOLOv5 algorithm. The YOLOv5 model can efficiently identify the position of the goods and price tags in the image. At the same time, for the problem of high-resolution small target detection, the folding head mechanism is introduced to optimize the YOLOv5 model to process high-resolution images and improve the accuracy of positioning.
[0057] Step S3: Based on the small image output by the target detection module, an image classification model is trained to classify the detected product area. For the small product image area detected by YOLOv5, the ResNet model is used for further classification. In view of the long tail and high similarity of product image data, the present invention optimizes the classification algorithm as follows:
[0058] 1) Construct a metric learning model: The classification problem is transformed into a metric learning problem, and the feature vector representation of each product category is trained through the optimized cosine Softmax loss function, so that the model can distinguish product packaging with extremely high similarity.
[0059] 2) Model optimization: Category-level additive margin is introduced into the loss function, so that the image classification model can enhance the distinction between different categories by adjusting the interval boundary of each category when performing category classification, and improve the model's recognition ability for long-tail and similar categories. For categories with high similarity, the cosine Softmax loss function increases the distance between such similar categories, thereby improving the classification accuracy. On the other hand, for categories that are easy to distinguish, the model will automatically reduce the loss between categories to avoid overfitting. The loss function is as follows:
[0060]
[0061] Loss function L CAMS It consists of two parts: negative sample loss and positive sample loss.
[0062] 1. Negative sample loss:
[0063] 1) This part of the loss aims to minimize the loss of negative samples (i.e., samples of different categories from the current sample).
[0064] 2) For each negative sample category c, calculate its difference from the current sample x i loss.
[0065] 3) Use minimize operation To adjust the perturbation vector Minimize the loss of negative samples.
[0066] 4) By adjusting ε c , the interval between different categories can be controlled, thereby enhancing the model's ability to distinguish different categories.
[0067] 2. Positive sample loss:
[0068] 1) This part of the loss aims to maximize the loss of positive samples (i.e. samples of the same category as the current sample).
[0069] 2) For each positive sample category c, calculate its difference from the current sample x i loss.
[0070] 3) Use the maximize operation To adjust the perturbation vector Maximize the loss of positive samples.
[0071] 4) In this way, it can ensure that the model has a higher recognition rate for samples of the same category.
[0072] Formula Symbols
[0073] L CMS: CAMS loss function, used to optimize the model.
[0074] n a : The number of categories in the negative sample set.
[0075] N a : A set of negative samples, that is, a set of samples of different categories from the current sample.
[0076] c: The category to which the current sample belongs.
[0077] S c : The set of samples belonging to the category.
[0078] x i : Sample i.
[0079] θ yi : Category y of sample i i The feature vector of .
[0080] ε c : The additive margin of category c, used to adjust the boundaries between categories.
[0081] Perturbation vector of category c, used to adjust the feature vector.
[0082] The norm of the perturbation vector does not exceed the additive margin ε c .
[0083] s: Scaling factor, used to control the influence of the interval.
[0084] nP a : Positive sample set P a The number of categories in .
[0085] P a : A set of positive samples, that is, a set of samples of the same category as the current sample.
[0086] cosθ j : The cosine similarity between the feature vector of category j and the feature vector of sample i.
[0087] The category-level perturbation bound is as follows: It is part of the Category-Level Additive Margin (CAMS) method for metric learning. The purpose of this perturbation bound is to dynamically adjust the margin for each category to improve the model's ability to distinguish similar categories.
[0088]
[0089] ε c : Class-level perturbation boundary for class c.
[0090] ε: Base interval, a constant used to adjust the initial intervals of all categories.
[0091] △ε: The amount of change in the interval, used to adjust the interval based on the distribution of the categories.
[0092] q c : The average number of samples of category c.
[0093] q1: The average number of samples of the category with the largest number of samples.
[0094] τ: A threshold used to distinguish between classes with more and fewer samples.
[0095] Step S4: Output of classification results. In practical applications, when a new product image is input, the product location in the image is first identified through the object detection model, and the feature vector of the product is generated through the classification model. Then, the feature vector is compared with the pre-stored vectors in the Milvus vector library for similarity to obtain the final classification result.
[0096] Through this optimization method, the present invention can effectively improve the accuracy and robustness of commodity display recognition in supermarket retail terminals, especially when facing complex scenes and similar commodity categories, it can still maintain efficient and accurate recognition performance and also improve the generalization ability of the model.
[0097] The YOLOv5 target detection algorithm and Resnet classification model are optimized to perform well in processing high-resolution images and small target detection, providing high-precision product positioning and classification results. Especially in the case of long-tail distribution and similar product categories, the present invention further improves the recognition accuracy and robustness of the model by introducing metric learning and loss function adjustment.
[0098] At the same time, this method has good scalability and can adapt to commodity recognition tasks of different types and scales, providing wide application support for retail terminals.
[0099] The embodiment of the present invention further provides a system for identifying commodity displays at retail terminals, which can be used to implement the above-mentioned method for identifying commodity displays at retail terminals. The system includes:
[0100] A data acquisition module is configured to collect image data of commodity displays at retail terminals, and then manually annotate and preprocess the data to obtain preprocessed data;
[0101] The object detection model training module is configured to train an object detection model using the YOLOv5 algorithm based on the image position information in the preprocessed data to detect the location of the product and price tag in the image;
[0102] The image classification model training module is configured to train an image classification model based on the small image output by the object detection module to classify the detected product areas;
[0103] The output module is configured to compare the vector obtained after the product image passes through the detection and image classification model with the Milvus vector library to obtain the final classification result.
[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and the aforementioned module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0105] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.
Claims
1. A method for identifying commodity displays at a retail terminal, characterized in that: include: S1, collect image data of commodity displays at retail terminals, and then manually annotate and preprocess the data to obtain preprocessed data; S2, based on the image position information in the preprocessed data, uses the YOLOv5 algorithm to train an object detection model to detect the location of the product and price tag in the image; S3, based on the small image output by the object detection module, train an image classification model to classify the detected product areas; S4, the vector obtained after the product image passes through the detection and image classification model is compared with the Milvus vector library to obtain the final classification result.
2. The method for identifying commodity displays at retail terminals according to claim 1, characterized in that: The S1 specifically includes: S11, collect image data of retail terminal commodity displays of different brands, different packaging forms, different shooting angles, and different lighting conditions; S12, preprocessing the image data, including adjusting the resolution of the image data, cutting out the background noise area, and normalizing the image pixels.
3. The method for identifying commodity displays at retail terminals according to claim 1, characterized in that: After S1 and before S2, the method further includes: For the collected image data, the following data augmentation strategies are adopted: (1) Randomly rotate the image by a certain angle to simulate different shooting angles; (2) Randomly scale images and crop unnecessary areas to enhance the model’s adaptability to different sizes and scenes; (3) Randomly adjust the brightness, contrast, and saturation of the image to adapt to different lighting conditions; (4) Randomly add noise to increase data diversity and prevent model overfitting.
4. The method for identifying commodity displays at retail terminals according to claim 1, characterized in that: The S2 specifically includes: During the training process, a folding head mechanism is introduced to address the problem of detecting small objects with high resolution.
5. The method for identifying commodity displays at retail terminals according to claim 1, characterized in that: The S3 specifically includes: S31, transforms the classification problem into a metric learning problem, and trains the feature vector representation of each product category through the optimized cosine Softmax loss function, so that the model can distinguish product packaging with extremely high similarity; S32, introduces category-level additive margin into the loss function, so that the image classification model can enhance the distinction between different categories by adjusting the interval boundary of each category when performing category classification, thereby improving the model's recognition ability for long-tail and similar categories.
6. The method for identifying commodity displays at retail terminals according to claim 5, characterized in that: The loss function of the image classification model is as follows: Among them, L CMS is the loss function used to optimize the model, nN a is the number of categories in the negative sample set, N a is a negative sample set, i.e., a sample set of different categories from the current sample, c is the category to which the current sample belongs, S c is the sample set belonging to the category, x i is sample i, is the category y of sample i i The eigenvector of c is the additive interval of category c, used to adjust the boundaries between categories, is the perturbation vector of category c, used to adjust the feature vector, The norm of the perturbation vector does not exceed the additive interval ε c , s is the scaling factor, which is used to control the influence of the interval, nP a is the positive sample set P a The number of categories in P a is the positive sample set, that is, the sample set of the same category as the current sample, cosθ j is the cosine similarity between the feature vector of category j and the feature vector of sample i.
7. The method for identifying commodity displays at retail terminals according to claim 1, characterized in that: The S4 specifically includes: When a new product image is input, the target detection model is used to identify the product location in the image, and the classification model is used to generate a feature vector for the product. Then, the feature vector is compared with the pre-stored vectors in the Milvus vector library for similarity to obtain the final classification result.
8. A commodity display identification system for retail terminals, characterized in that: The system can be used to implement the method for identifying commodity displays at retail terminals as described in any one of claims 1 to 7, and the system includes: A data acquisition module is configured to collect image data of commodity displays at retail terminals, and then manually annotate and preprocess the data to obtain preprocessed data; The object detection model training module is configured to train an object detection model using the YOLOv5 algorithm based on the image position information in the preprocessed data to detect the location of the product and price tag in the image; The image classification model training module is configured to train an image classification model based on the small image output by the object detection module to classify the detected product areas; The output module is configured to compare the vector obtained after the product image passes through the detection and image classification model with the Milvus vector library to obtain the final classification result.
Citation Information
Cited By
Testing device and testing method for light supplementing lamp of networked toll portal
CN121917202A