A method and system for inventory counting of cigarette retail merchants

Abstract

The present invention belongs to the field of electronic information technology for image processing, and discloses a method and system for inventory checking of cigarette retail merchants. The inventory checking method includes using an image detection model to perform object detection on the collected inventory cigarette images, obtaining prediction boxes containing target images in the inventory cigarette images and obtaining each target area image; using a text recognition model to perform product specification keyword recognition on each target area image, if keywords are recognized, obtaining a product specification classification result, if not recognized, inputting the target area image into an image classification module; then using an image classification model to classify the input target area image to obtain a product specification classification result; inputting the position information and the product specification classification result into a statistical model for statistics to obtain the inventory checking result of the cigarette retail merchant. The inventory checking method of the present invention can intelligently, quickly, accurately and effectively identify and check a wide variety of cigarette inventories, saving labor costs and providing data support for enterprise operation.

Description

Technical Field

[0001] The present invention belongs to the field of electronic information technology for image processing, and particularly relates to a method and system for inventory checking of cigarette retail merchants. Background Art

[0002] In order to win consumers' choices in a huge number of offline terminals, compete for the exposure position and proportion of products on the shelves, determine which competing products to be exposed together with, and how to make the visual presentation of products more attractive to consumers through shelf display, it has currently become the top priority of the terminal execution of fast-moving consumer goods enterprises. In addition, when out-of-stock situations occur on the shelves, brand owners may lose 46% of buyers, and retailers may lose 30% of buyers. By using AI item recognition technology to assist manual inventory review, expensive labor costs can be bypassed, and efficient management of offline terminals can be achieved, enabling managers to more closely control in-store operations and obtain competitive business insights through data analysis to identify business opportunities at all sales points. AI item recognition technology can view, identify, and process images and videos like the human eye, and its capabilities can assist in various tasks, including signal processing, image enhancement, object detection and classification, motion analysis, and 3D image reconstruction. Currently, global brands such as Coca-Cola, Anheuser-Busch InBev, Heineken, Nestle, and Henkel have all used this technology for digital upgrading of retail terminals, but there is still a blank in the inventory recognition system applied to cigarettes.

[0003] In order to better grasp and manage cigarette retail merchants, it is necessary to master comprehensive and real-time terminal inventory, display, and price data. However, in the current management methods, the following problems exist: First, the sample quantity is missing. Although the information collection mode can adopt a combination of human and machine, it is still impossible to ensure that every order is scanned and recorded truthfully. Second, manual inventory checking is inefficient, costly, and the quality of manual inventory data is difficult to measure. Third, the data analysis dimension is single, and it cannot provide comprehensive data support for terminal social inventory management. On the one hand, after the data is entered, it is not stored in a structured manner, making it difficult to output standardized reports. On the other hand, manual collection can only provide item and quantity data, and other terminal data cannot be restored (such as the display position and display method of each item), thus providing more comprehensive data analysis indicators. Moreover, researching the digital management system for social inventory has important practical significance: 1. Improve the integrity and authenticity of social inventory data; 2. Optimize the business model and promote the deep integration of big data and business; 3. Gain insights into consumers' real needs and promote supply optimization from the demand side; 4. Provide theoretical support for research in related fields and promote the development of research in this area. Therefore, with the increasing urgency of the demand for refined inventory management and the increasingly widespread application of AI technology, it is necessary to reflect on the existing social inventory management model and rely on innovative technologies to improve management effectiveness. Summary of the Invention

[0004] In view of the problems and deficiencies existing in the prior art, the purpose of the present invention is to provide a method and system for inventory checking of cigarette retail merchants.

[0005] Based on the above purpose, the present invention adopts the following technical solutions:

[0006] The first aspect of the present invention provides a method for inventory checking of cigarette retail merchants, including the following steps:

[0007] S1: Collect multiple inventory cigarette images of cigarette retail merchants;

[0008] S2: Use an image detection model to perform object detection on the inventory cigarette images collected in step S1, obtain prediction frames containing cigarette target images in the inventory cigarette images, and further obtain each cigarette target region image and the position information of the cigarette target region image;

[0009] S3: Use a text recognition model to recognize the cigarette specification text keywords in each cigarette target region image. If the cigarette specification keywords are recognized, obtain the specification classification result of the cigarette target region image; if the cigarette specification keywords are not recognized, input the cigarette target region image into an image classification model;

[0010] S4: Use an image classification model to classify the input cigarette target region image to obtain the specification classification result of the cigarette target region image;

[0011] S5: Input the position information of each cigarette target region image obtained in step S2 and the specification classification results of each cigarette target region image obtained in steps S3 and S4 into a statistical model to count the position, specification and quantity of cigarettes, and obtain the inventory checking result of cigarette retail merchants.

[0012] More preferably, before inputting the inventory cigarette images collected in step S1 into the image detection model, first input them into an image processing model for reflection recognition and removal; the image processing model is Reflection-free Flash-only Cure.

[0013] More preferably, the image detection model regards cigarettes of all specifications as the same category for detection to obtain the position regions of all cigarette targets.

[0014] Preferably, the image detection model adopts a trained Yolov4 object detection model.

[0015] Preferably, the Yolov4 object detection model specifically includes: an input layer, a BackBone backbone network, an insertion layer Neck, and an output layer Prediction; among them, the input layer includes Mosaic data augmentation, cmBN, and SAT self-adversarial training; the BackBone backbone network includes CSPDarknet53, Mish activation function, and Dropblock; the insertion layer Neck includes CBL, SPP module, and FPN+PAN structure; the output layer Prediction includes an anchor box mechanism, a loss function during training, and DIOU_nms for predicting box screening; the loss function during training includes a bounding box regeession loss function, a confidence loss function, and a classification loss function, where the bounding box regression loss function uses CIOU_Loss, the confidence loss function uses a cross-entropy loss function based on the Bounding Box, and the classification loss uses a binary cross-entropy loss function based on IOU.

[0016] More preferably, the training process of the Yolov4 object detection model includes the following steps:

[0017] (1) Collect multiple inventory cigarette images of cigarette retail merchants, calibrate the target information in the inventory cigarette images, and establish an inventory cigarette image set with the calibrated inventory cigarette images; randomly divide the inventory cigarette image set into a training set, a validation set, and a test set according to a ratio of 8:1:1; the target information includes at least the position and classification information of the target cigarette; the training set images are data-augmented according to the actual scenario, and the data augmentation methods include brightness adjustment, rotation, cropping, and blurring.

[0018] (2) Use the training set to train the object detection model, set the input size of the model to 608*608, select 3 different sets of hyperparameter combinations for model training, update the object detection model, and obtain the trained object detection model.

[0019] (3) Use the validation set to preliminarily verify the models trained with 3 different sets of hyperparameters, verify the generalization ability of the models, select 1 set of the best-performing model hyperparameters, and continue model training. Stop model training when the performance on the training set no longer improves.

[0020] (4) Use the test set to test the trained object detection model obtained in step (3), select the optimal object detection model from the trained object detection models, and obtain the trained Yolov4 object detection model.

[0021] More preferably, the image classification model is used to solve the problem of sample imbalance between cigarette specifications.

[0022] Preferably, the image classification module uses a trained ResNet50 image classification model, and the Focal Loss function is selected during training.

[0023] More preferably, the specific process of using the image classification model to classify the input cigarette target area image is as follows: the input cigarette target area image is input into the ResNet-50 network to obtain the output result of the cigarette target area image at the Conv5 layer of the ResNet-50 network. The output result contains the feature vector of a single cigarette target area image, and then the output feature vector is input into the fully convolutional network for classification to obtain the product specification classification result of the cigarette target area image.

[0024] More preferably, the training process of the image classification module includes the following steps:

[0025] (A) Collect cigarette target images of different product specifications to establish a cigarette target image set, and store the target images in the cigarette target image set in different folders according to different product specifications; randomly divide the cigarette target image set into a training set, a validation set, and a test set according to a ratio of 8:1:1;

[0026] (B) Use the training set to train the ResNet50 image classification model, select 3 groups of different hyperparameter combinations for model training, update the image classification model, and obtain the trained image classification model;

[0027] (C) Use the validation set to preliminarily verify the models trained with 3 groups of different hyperparameters, verify the model generalization ability, select 1 group of the best-performing model hyperparameters, and continue model training. Stop model training when the performance on the training set no longer improves;

[0028] (D) Use the test set to test the trained image classification model obtained in step (C), select the optimal image classification model from the trained image classification models, and obtain the trained ResNet50 image classification model.

[0029] Preferably, the character recognition model uses the OCR recognition algorithm to extract text information from the cigarette target area image detected in step S2, and then uses the regular matching classification model to perform regular matching recognition on the extracted text information and the cigarette product specification text keyword information: if the cigarette product specification keyword is recognized, the product specification classification result of the cigarette target area image is obtained; if the cigarette product specification keyword is not recognized, the cigarette target area image is input into the image classification model; the regular matching classification model is constructed based on the cigarette product specification text keyword information.

[0030] Preferably, the statistics module uses the DBSCAN algorithm to cluster and count the position information of the cigarette target area image, and obtains the position inventory result of the cigarette inventory; the statistics module uses the category statistics algorithm to count the product specification classification result and quantity information of the cigarette target area image, and obtains the product specification and quantity inventory result of the cigarette inventory.

[0031] The second aspect of the present invention provides a cigarette inventory checking system, which includes a human-computer interaction subsystem and a data processing subsystem;

[0032] The human-computer interaction subsystem includes a human-computer interaction interface and an image acquisition module; the human-computer interaction interface is used to provide an operation interface for the user to input the inventory cigarette images or videos of the cigarette merchant collected into the system; the image acquisition module is used to collect the inventory cigarette images or videos of the cigarette merchant, and upload the collected inventory cigarette images or videos to the data processing subsystem;

[0033] The data processing subsystem includes an inventory cigarette image recognition module and a cigarette inventory checking module; the inventory cigarette image recognition module includes any of the image detection models, character recognition models, and image classification models in the first aspect above, and is used to identify the cigarette targets in the uploaded inventory cigarette images and output the recognition results. The cigarette target recognition results at least include the product specifications and position information of the cigarettes; the cigarette inventory checking module includes any of the statistical models in the first aspect above, and is used to count the cigarette information recognition results to obtain the inventory checking results of the cigarette merchant.

[0034] Preferably, the human-computer interaction interface further includes a merchant information recognition module, and the merchant information recognition module is used to identify merchant information by using an OCR recognition model.

[0035] More preferably, the human-computer interaction subsystem further includes a merchant management module, and the merchant management module is used to manage tasks, rules, record information, etc. of users and merchants.

[0036] The merchant management module includes a store information management sub-module, a store task management sub-module, a store visit personnel management sub-module, a store visit record management sub-module, a store rule management sub-module, a user management sub-module, and a monopoly confiscated cigarette warehouse management sub-module. The store information management sub-module is used to manage store information and count store lists. The store task management sub-module is used to manage store tasks and publish announcement information. The store visit personnel management sub-module is used to manage store visit personnel information. The store visit record management sub-module is used to manage record information such as store check-ins of store visit personnel. The store rule management sub-module is used to manage the mandatory distribution rules of stores and the verification rules of promotional activities. The user management sub-module is used to manage the permissions of users to log in and / or use the system. The monopoly confiscated cigarette warehouse management sub-module is used to manage monopoly confiscated warehouse information and count warehouse lists.

[0037] More preferably, the human-computer interaction interface further includes a store visit personnel information recognition module, which is used to recognize the store visit information recorded by the store visit personnel by using an OCR recognition model.

[0038] The third aspect of the present invention provides an electronic device, including a memory and a processor, and a computer program is stored on the memory. It is characterized in that when the processor executes the computer program, it implements the cigarette retail merchant inventory checking method as described in any one of the first aspects above, and / or loads the cigarette inventory checking system as described in any one of the second aspects above.

[0039] The fourth aspect of the present invention provides a computer-readable storage medium, which is characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the cigarette retail merchant inventory checking method as described in any one of the first aspects above, and / or loads the cigarette inventory checking system as described in any one of the second aspects above.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] (1) In this application, the trained Yolov4 object detection model is first used to detect all cigarette specifications as one category to obtain the position areas of all cigarette targets. Then, the trained ResNet50 deep learning classification model is combined with the Focal Loss function to support the fine-grained recognition of similar cigarette items, enabling the neural network to better "focus" on the subtle differences, and thus better distinguish similar objects, solving the problem of sample imbalance between cigarette specifications. In response to the characteristics of cigarette inventory and the problems that occur in existing algorithms during detection, such as a large variety of cigarette types (more than 600 types) and the positive and negative sample imbalance characteristics that occur in the Yolov4 object detection algorithm among a large number of targets. In addition, cigarettes are densely placed, the boundaries of cigarettes are not easy to distinguish, there is reflection and low light, and there are occlusions, etc., resulting in difficulties in object detection, false detection, or missed detection, thus leading to low detection accuracy. Therefore, it is necessary to improve the existing object detection algorithm to achieve the detection of pictures or videos of cigarette inventory, and further detect information such as the position, specification, and quantity of cigarettes in the pictures or videos. Thus, the problem of object detection in the scenario of densely placed target objects is specifically solved, such as the anchor box overlap caused by highly overlapping instances and the incorrect suppression of prediction results by NMS. In one of the embodiments, the inventory cigarette image detection model of this application respectively identifies 50 groups of box cigarette key specification pictures and 50 groups of carton cigarette key specification pictures randomly selected. The average detection rates of the object detection model are 99.35% and 99.15% respectively, and the average accuracies of the image classification model are 98.33% and 97.35% respectively. In addition, the inventory cigarette image classification model of this invention supports the recognition of 165 cigarette specifications. For other non-key specifications, the average recognition accuracy is not less than 90%, and the recognition of the cigarette inventory of a retail merchant (within 30 inventory photos) is completed within no more than 3 minutes. Therefore, the inventory cigarette image recognition module of this invention can quickly, accurately, and effectively detect and identify cigarette inventory.

[0042] (2) Before the image detection model of this invention, a cabinet glass reflection recognition and removal module based on Reflection-free Flash-only Cure (RFC) is also added to effectively restore the original structure of the photo and the appearance characteristics of the commodity, improving the detection accuracy.

[0043] (3) This application also combines with the existing data collection system (image acquisition module) for retail merchants, adds an upload interface for inventory cigarette images or videos, enables the inventory cigarette images or videos to be uploaded to the system, and at the same time uses the inventory cigarette image recognition module proposed in this application to establish a cigarette inventory counting system. The cigarette inventory counting system of this application takes the inventory cigarette images or videos uploaded by retail merchants as input, and inputs the detected cigarette specifications and location information output by the inventory cigarette image recognition module into the cigarette inventory counting module for statistics and recording into the inventory of this merchant, and then transmits the inventory information of the merchant to data analysts for further statistical processing. Based on the rapid, accurate and effective recognition of cigarette inventory using the inventory cigarette image recognition module, this application conducts intelligent statistical inventory counting, which not only effectively makes up for the problem of missing statistical data, but also solves the problems of low efficiency, high cost and difficult control of statistical data quality in manual inventory counting. It can online and real-time master the basic information of cigarette products, accurately restore information such as the variety, quantity, display location, display method of cigarettes, and provide favorable data support for the operation and service of the cigarette industry using more comprehensive data analysis indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flowchart of the method for counting the inventory of cigarette retail merchants of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the following further elaborates on the present invention in detail through embodiments in conjunction with the drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the drawings and combine the embodiments to elaborate on this application in detail.

[0047] Embodiment 1

[0048] The embodiment of the present invention provides a method for counting the inventory of cigarette retail merchants, including the following steps:

[0049] S1: Use an image acquisition module to collect multiple inventory cigarette images of cigarette retail merchants; then input the inventory cigarette images into an image processing model for specular reflection recognition and removal to obtain processed inventory cigarette images; the image processing model is Reflection-free Flash-only Cure.

[0050] S2: Use the image detection model in the image recognition module to perform object detection on the inventory cigarette image processed in step S1, obtain the prediction boxes containing cigarette target images in the inventory cigarette image, and further obtain each cigarette target region image and the position information of the cigarette target region image.

[0051] Since cigarette images belong to a dense target scenario, in deep learning, low-dimensional feature maps contain low-level semantic information, but have more accurate coordinate information for small targets; high-dimensional feature maps contain higher-level semantic information and have better responses for large targets. Therefore, the YOLOv4 object detection model is used to detect the cigarette scenario. However, due to the large variety and imbalance of cigarette types, the target of the YOLOv4 object detection model is set to only perform single-object detection on cigarette instances, and cigarettes of all specifications are combined into one category for detection, that is, only the instance positions of cigarettes are detected, and the specifications are not distinguished, so as to remove the impact of sample imbalance on the missed detection of cigarette instances.

[0052] The construction process of the YOLOv4 object detection model includes: using the TensorFlow deep learning framework to create the backbone of the object detection network. The backbone network is CSPDarkNet53, which contains 29 convolutional layers, a receptive field of 725*725, and 27.6M parameters. It draws on CSPNet to solve the problem of repeated gradient information in network optimization in other large convolutional neural network frameworks' backbones, integrating the changes in gradients into the feature maps from beginning to end, thus reducing the number of model parameters and FLOPS values, ensuring both inference speed and accuracy while reducing the model size; The loss function of the YOLOv4 object detection model is divided into three parts: 1) bounding box regression loss, 2) confidence loss, 3) classification loss; among which the bounding box regression loss uses the CIOU loss L CIOU = 1 - IOU(A, B)+ρ 2 (A ctr , B ctr ) / c 2 +α.υ; The aspect ratio settings of the anchor boxes are generated by clustering the labeled data set, so that the features of the data in the actual scenario can be better extracted during training. Usually, there are 9 types of anchor boxes, and the sizes of the anchor boxes are different on feature maps of different scales; An improved version of DIOU_NMS is used for the screening of prediction boxes.

[0053] Train the YOLOv4 object detection model. The training process includes the following steps:

[0054] (1) Collect multiple inventory cigarette images of cigarette retail merchants, and use the LabelIMG dataset calibration tool to calibrate the position and classification information in the inventory cigarette images. Establish an inventory cigarette image set with the calibrated inventory cigarette images; randomly divide the inventory cigarette image set into a training set, a validation set, and a test set according to a ratio of 8:1:1; the target information includes at least the position and classification information of the target cigarette; the training set images are data-augmented according to the actual scenario, and the data augmentation methods include brightness adjustment, rotation, cropping, and blurring;

[0055] Furthermore, retail merchants or customer managers can use shooting devices such as cameras and mobile phones to take pictures of the inventory cigarettes. It is required that the pictures be as clear as possible, cover comprehensively, have no occlusion in the front and back of the cigarettes, and try to ensure that pictures are taken of different merchants, different backgrounds, different specifications, different quantities, and different angles to ensure the sample diversity of different specification cigarettes in the image set. Preferably, the inventory cigarette image set requires that the number of pictures is not less than 1000, the number of occurrences of each specification cigarette (i.e., the single-specification sample quantity) is not less than 1000, and the number of cigarette merchants collected is not less than 50.

[0056] (2) Use the training set to train the target detection model. Set the input size of the model to 608*608, select 3 groups of different hyperparameter combinations for model training, update the target detection model, and obtain the trained target detection model;

[0057] (3) Use the validation set to preliminarily verify the models trained with 3 groups of different hyperparameters, verify the generalization ability of the models, select 1 group of the best-performing model hyperparameters, and continue model training. Stop model training when the performance on the training set no longer improves;

[0058] (4) Use the test set to test the trained target detection model obtained in step (3), select the optimal target detection model from the trained target detection models, and obtain the trained Yolov4 target detection model.

[0059] Furthermore, the image detection model uses the trained Yolov4 target detection model.

[0060] S3: Use the text recognition model to recognize the cigarette specification text keywords for each cigarette target area image. The recognition process is as follows:

[0061] Use the OCR recognition algorithm in the text recognition model to extract text information from the cigarette target area image detected in step S2, and then use the regular matching classification model to perform regular matching and recognition on the extracted text information and the cigarette product specification text keyword information: If the cigarette product specification keyword is recognized, the product specification classification result of the cigarette target area image is obtained; if the cigarette product specification keyword is not recognized, the cigarette target area image is input into the image classification model.

[0062] The regular matching classification model is constructed based on the cigarette product specification text keyword information. Further, according to the cigarette product specification information, a regular matching classification model based on text keywords is constructed to facilitate regular matching of the text information extracted by OCR to identify the cigarette product specification. If the product specification keyword is matched, the cigarette product specification is recognized; if the product specification keyword is not matched, the image classification model is continued to be used to classify and identify the cigarette.

[0063] S4: Use the image classification model to classify the input cigarette target image to obtain the product specification classification result of the cigarette target area image.

[0064] The image classification module model uses the trained ResNet50 deep learning image classification model. When training, the Focal Loss function is selected to solve the problem of sample imbalance between cigarette product specifications. The specific process of using the image classification module for the input cigarette target image is: input the input cigarette target image into the ResNet-50 network to obtain the output result of the cigarette target image in the Conv5 layer of the ResNet-50 network. The output result contains the feature vector of a single cigarette target area image, and then the output feature vector is input into the fully convolutional network for classification to obtain the product specification classification result of the cigarette target area image.

[0065] The training process of the image classification module includes the following steps:

[0066] (A) Collect cigarette target images of different product specifications to establish a cigarette target image set, and store the target images in the target image set in different folders according to different product specifications; randomly divide the cigarette target image set into a training set, a validation set, and a test set according to a ratio of 8:1:1;

[0067] (B) Use the training set to train the ResNet50 image classification model, select 3 groups of different hyperparameter combinations for model training, update the image classification model, and obtain the trained image classification model;

[0068] (C) Use the validation set to preliminarily validate the models trained with three groups of different hyperparameters, verify the generalization ability of the models, select the best-performing group of model hyperparameters, and continue with model training. Stop the model training when the performance on the training set no longer improves;

[0069] (D) Use the test set to test the trained image classification model obtained in step (C), select the optimal image classification model from the trained image classification models, and obtain the trained ResNet50 image classification model.

[0070] S5: Input the position information of each cigarette target area image obtained in step S2 and the product specification classification results of each cigarette target area image obtained in steps S3 and S4 into the statistical model, and count the positions, product specifications, and quantities of the cigarettes to obtain the inventory count results of the cigarette retail merchants.

[0071] The statistical module uses the DBSCAN algorithm to cluster and count the position information of the cigarette target area images to obtain the inventory location count results of the cigarettes; the statistical module uses the category counting algorithm to count the product specification classification results and quantity information of the cigarette target area images to obtain the product specification and quantity count results of the cigarette inventory.

[0072] Embodiment 2

[0073] A cigarette inventory counting system includes a human-computer interaction subsystem and a data processing subsystem.

[0074] The human-computer interaction subsystem includes a human-computer interaction interface, an image acquisition module, and a merchant management module; the human-computer interaction interface is used to provide an operation interface for users to input the inventory cigarette images or videos of the cigarette merchants collected into the system; the image acquisition module is used to collect the inventory cigarette images or videos of the cigarette merchants and upload the collected inventory cigarette images or videos to the data processing subsystem; the merchant management module is used to manage tasks, rules, record information, etc. of users and merchants.

[0075] The human-computer interaction interface includes a merchant information recognition module and a store visit personnel information recognition module; the merchant information recognition module is used to recognize merchant information using an OCR recognition model, where the merchant information includes but is not limited to the merchant's business name, business license, and tobacco monopoly license number, etc.; the store visit personnel information recognition module is used to recognize the store visit information recorded by the store visit personnel using an OCR recognition model.

[0076] The image acquisition module includes an existing commodity data acquisition system for retail merchants and adds an upload interface for pictures or videos so that the inventory cigarette images or videos can be uploaded to the system.

[0077] The merchant management module includes a store information management sub-module, a store task management sub-module, a store visit personnel management sub-module, a store visit record management sub-module, a store rule management sub-module, a user management sub-module, and a confiscated tobacco warehouse management sub-module for monopolized sales. The store information management sub-module is used to manage store information and count the store list. The store task management sub-module is used to manage store tasks and publish announcement information. The store visit personnel management sub-module is used to manage the information of store visit personnel. The store visit record management sub-module is used to manage record information such as the store check-in of store visit personnel. The store rule management sub-module is used to manage the mandatory distribution rules of stores and the verification rules of promotional activities. The user management sub-module is used to manage the permissions for users to log in to and / or use the system. The confiscated tobacco warehouse management sub-module for monopolized sales is used to manage the information of the confiscated tobacco warehouse for monopolized sales and count the warehouse list.

[0078] The data processing subsystem includes an inventory cigarette image recognition module and a cigarette inventory counting module; the inventory cigarette image recognition module includes the image detection model, character recognition model, and image classification model described in Embodiment 1, and is used to identify cigarette targets in the uploaded inventory cigarette images and output the recognition results. The cigarette target recognition results at least include the product specifications and location information of the cigarettes. The cigarette inventory counting module includes the statistical model described in Embodiment 1 and is used to count the cigarette information recognition results to obtain the inventory counting results of cigarette merchants.

[0079] Embodiment 3

[0080] An electronic device includes a memory and a processor. A computer program is stored on the memory. When the processor executes the computer program, any step in the cigarette retail merchant inventory counting method described in Embodiment 1 is implemented, and / or the cigarette inventory counting system described in Embodiment 2 is loaded.

[0081] Embodiment 4

[0082] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, any step in the cigarette retail merchant inventory counting method described in Embodiment 1 is implemented, and / or the cigarette inventory counting system described in Embodiment 2 is loaded.

[0083] The cigarette retail merchant inventory counting method described in Embodiment 1 of the present invention is used to perform SKU target detection recognition and product specification classification recognition on 50 randomly selected pictures of key product specifications of box cigarettes and 50 pictures of key product specifications of carton cigarettes respectively, and calculate the detection rate of the image detection model and the classification accuracy rate of the image classification model. Among them, the detection rate = total number of accurately detected SKUs / total actual SKUs, and the classification accuracy rate = number of correctly recognized / number of accurately detected SKUs. The results are shown in Table 1-4.

[0084] Table 1 Target detection results of pictures of 50 randomly selected key cigarette SKUs in cartons

[0085] Total actual SKUs Total detected SKUs Total accurately detected SKUs SKU detection rate SKU precision rate 2315 2320 2300 99.35％ 99.14％

[0086] Table 2 Classification recognition results of pictures of 50 randomly selected key cigarette SKUs in cartons

[0087]

[0088]

[0089]

[0090] Table 3 Target detection results of pictures of 50 randomly selected key cigarette SKUs in packs

[0091] Total actual SKUs Total detected SKUs Total accurately detected SKUs SKU detection rate SKU precision rate 1182 1190 1172 99.15％ 98.49％

[0092] Table 4 Classification recognition results of pictures of 50 randomly selected key cigarette SKUs in packs

[0093]

[0094]

[0095]

[0096] As can be seen from Tables 1-4, the optimal inventory cigarette image detection model trained in the present invention respectively identifies 50 randomly selected pictures of key cigarette SKUs in cartons and 50 pictures of key cigarette SKUs in packs. The average detection rates of the target detection model are 99.35% and 99.15% respectively, and the average accuracies of the image classification model are 98.33% and 97.35% respectively. In addition, the inventory cigarette image classification model of the present invention supports the identification of 165 cigarette SKUs. For other non-key SKUs, the average recognition accuracy is not less than 90%, and the time required to complete the identification of the cigarette inventory of a retail merchant (within 30 inventory photos) does not exceed 3 minutes. The inventory cigarette image detection model of the present invention also supports the recognition of reflective glass in the floor cabinet. By analyzing the reflective characteristics, it can extract a separate reflective layer and remove it, effectively restoring the original structure of the photo and the appearance characteristics of the commodity, and improving the recognition accuracy; it supports the fine-grained recognition of similar cigarette items. Based on common image recognition networks, different auxiliary mechanisms are introduced to enable the neural network to better "focus" on the subtle differences, and thus better distinguish similar objects. Therefore, the inventory cigarette image detection model of the present invention can quickly, accurately and effectively identify the cigarette inventory.

[0097] In summary, the present invention effectively overcomes the deficiencies in the prior art and has high industrial utilization value. The role of the above embodiments is to illustrate the substantial content of the present invention, but does not limit the protection scope of the present invention. Those of ordinary skill in the art should understand that the technical solution of the present invention can be modified or equivalently replaced without departing from the essence and protection scope of the technical solution of the present invention.

Claims

1. A method for inventory checking of cigarette retail merchants, characterized in that, It includes the following steps: S1: Collect multiple inventory cigarette images of cigarette retail merchants; S2: Use an image detection model to perform object detection on the inventory cigarette images collected in step S1, obtain prediction boxes containing cigarette target images in the inventory cigarette images, and then obtain each cigarette target area image and the position information of the cigarette target area image; S3: Use a text recognition model to identify the cigarette product specification text keywords in each cigarette target area image. If the cigarette product specification keywords are recognized, obtain the product specification classification result of the cigarette target area image; If the cigarette product specification keywords are not recognized, input the cigarette target area image into an image classification model; S4: Use the image classification model to classify the input cigarette target area image to obtain the product specification classification result of the cigarette target area image; S5: Input the position information of each cigarette target area image obtained in step S2 and the product specification classification results of each cigarette target area image obtained in steps S3 and S4 into a statistical model, and count the position, product specification, and quantity of the cigarettes to obtain the inventory count result of the cigarette retail merchant; The text recognition model uses the OCR recognition algorithm to extract text information from the cigarette target area images detected in step S2, and then uses a regular matching classification model to perform regular matching recognition on the extracted text information and the cigarette product specification text keyword information: If the cigarette product specification keywords are recognized, obtain the product specification classification result of the cigarette target area image; If the cigarette product specification keywords are not recognized, input the cigarette target area image into an image classification model; The regular matching classification model is constructed based on the cigarette product specification text keyword information; The image detection model regards cigarettes of all product specifications as the same category for detection, and is used to obtain the position areas of all cigarette targets; The image detection model uses the trained Yolov4 object detection model; The image classification model uses the trained ResNet50 image classification model, and the Focal Loss function is selected during training; The statistical model uses the DBSCAN algorithm to cluster and count the position information of the cigarette target area images to obtain the position count result of the cigarette inventory; The statistical model uses a category counting algorithm to count the product specification classification results and quantity information of the cigarette target area images to obtain the product specification and quantity count results of the cigarette inventory.

2. A cigarette inventory checking system, characterized in that, It includes a human-computer interaction subsystem and a data processing subsystem; The human-computer interaction subsystem includes a human-computer interaction interface and an image acquisition module; The human-computer interaction interface is used to provide an operation interface for the user to input the inventory cigarette images or videos of the cigarette merchants collected into the system; The image acquisition module is used to collect the inventory cigarette images or videos of the cigarette merchants and upload the collected inventory cigarette images or videos to the data processing subsystem; The data processing subsystem includes an inventory cigarette image recognition module and a cigarette inventory counting module; The inventory cigarette image recognition module includes the image detection model, character recognition model, and image classification model described in claim 1, and is used to recognize cigarette targets in the uploaded inventory cigarette images and output recognition results. The cigarette target recognition results at least include the product specifications and location information of the cigarette. The cigarette inventory counting module includes the statistical model described in claim 1, and is used to count the cigarette information recognition results to obtain the inventory counting results of cigarette merchants.

3. The cigarette inventory counting system according to claim 2, wherein The human-computer interaction interface further includes a merchant information recognition module, and the merchant information recognition module is used to recognize merchant information by using an OCR recognition model.

4. An electronic device, comprising a memory and a processor, wherein a computer program is stored on the memory, characterized in that, When the processor executes the computer program, it implements the cigarette retail merchant inventory counting method described in claim 1, and / or loads the cigarette inventory counting system described in claim 2 or 3.

5. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, it implements the cigarette retail merchant inventory counting method described in claim 1, and / or loads the cigarette inventory counting system described in claim 2 or 3.

Images