A tobacco brand classification method and apparatus

Abstract

The application provides a tobacco brand classification method and device, relates to the technical field of visual identification, and solves the technical problem of low recognition accuracy of tobacco goods in the existing technology under the condition of dense stacking and local occlusion. The method specifically comprises the following steps: obtaining multiple tobacco images; performing target detection on the multiple tobacco images based on a target detection model to obtain a first target region; matching the first target region with a preset difficult-to-identify target library to determine a second target region; performing instance segmentation on the second target region based on an instance segmentation model to obtain a segmentation mask; performing deduplication counting based on the segmentation mask and obtaining a target subgraph with the background removed; performing fine-grained feature extraction on regions in the first target region except the second target region and the target subgraph based on a fine-grained feature extraction model to obtain a feature vector; and matching and searching the feature vector with a standard vector library to output a classification result. The application is used in tobacco brand classification.

Description

Technical Field

[0001] This application relates to the field of visual recognition technology, and in particular to a method and apparatus for classifying tobacco brands. Background Technology

[0002] The tobacco industry's inventory counting and asset management place high demands on the accuracy and efficiency of product identification. Current technologies typically employ manual counting or general image recognition tools for assistance. Manual counting relies on staff to physically count each pack, carton, and box of cigarettes and manually input the data into the system, which is time-consuming, labor-intensive, and prone to errors. General image recognition algorithms are not optimized for the diverse forms of cigarette packs, cartons, and boxes. In scenarios with dense stacking of goods, uneven lighting, or partial occlusion, the accuracy of product identification drops significantly, and the identification results cannot be integrated with the inventory asset ledger system. Therefore, existing technologies suffer from low accuracy in identifying tobacco products in densely stacked and partially occluded scenarios. Summary of the Invention

[0003] This application provides a tobacco brand classification method and apparatus, which solves the technical problem of low tobacco product recognition accuracy in scenarios with dense stacking and partial occlusion in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution: Firstly, a method for classifying tobacco brands is provided, comprising: acquiring multiple tobacco images; performing target detection on the multiple tobacco images based on a target detection model to obtain a first target region; matching the first target region with a preset difficult-to-identify target library to determine a second target region; performing instance segmentation on the second target region based on an instance segmentation model to obtain a segmentation mask; performing deduplication and counting based on the segmentation mask to obtain a target sub-image with background removed; performing fine-grained feature extraction on the regions of the first target region other than the second target region and the target sub-image based on a fine-grained feature extraction model to obtain a feature vector; and matching and retrieving the feature vector with a standard vector library to output the classification result.

[0005] In conjunction with the first aspect mentioned above, in one possible implementation, target detection is performed on multiple tobacco images based on a target detection model to obtain a first target region, including: inputting multiple tobacco images into a YOLOv11 target detection model; processing the multiple tobacco images through high-resolution adaptation and non-maximum suppression (NMS) parameter adjustment, and outputting the bounding boxes and detection confidence scores of all tobacco products; and cropping the bounding boxes to obtain the region of interest (ROI) sub-image of each tobacco product as the first target region.

[0006] In conjunction with the first aspect mentioned above, in one possible implementation, the first target region is matched with a preset difficult-to-identify target library to determine the second target region, including: obtaining a coarse classification result of the first target region; matching the coarse classification result with difficult-to-identify category tags in the preset difficult-to-identify target library; if the coarse classification result matches any difficult-to-identify category tag in the preset difficult-to-identify target library, then the corresponding first target region is determined as the second target region.

[0007] In conjunction with the first aspect mentioned above, in one possible implementation, deduplication counting based on the segmentation mask includes: calculating the minimum bounding rectangle of the corresponding target based on the segmentation mask; calculating the aspect ratio feature based on the minimum bounding rectangle; performing deduplication processing on the overlapping mask by combining the number of mask instances and non-maximum suppression (NMS); and completing the target counting based on the number of mask instances after deduplication.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, obtaining the target sub-image with the background removed includes: generating a guided cropping box based on the contour of the segmentation mask; cropping the second target region using the guided cropping box to remove background pixels; and using the cropped clean single-item image as the target sub-image.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, based on a fine-grained feature extraction model, fine-grained feature extraction is performed on the regions and target sub-images in the first target region other than the second target region to obtain feature vectors. This includes: using the DINO model to extract features from the regions and target sub-images in the first target region other than the second target region; performing L2 normalization on the extracted original feature vectors; filtering out invalid Region of Interest (ROI) features that are too small or completely black, and retaining the effective high-dimensional feature vectors as feature vectors.

[0010] In conjunction with the first aspect mentioned above, one possible implementation involves matching and retrieving the feature vector against a standard vector library to output classification results. This includes: calculating the similarity between the feature vector and each standard vector in the standard vector library; if the similarity is greater than or equal to a preset threshold, outputting the corresponding brand or specification classification result; if the similarity is less than the preset threshold, outputting an unknown category result; and for multiple candidate classification results with similarity differences less than a preset difference threshold, using the aspect ratio features corresponding to the segmentation mask for auxiliary disambiguation to determine the final classification result from the multiple candidate classification results.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, the second target region is segmented based on an instance segmentation model to obtain a segmentation mask, including: inputting the second target region into the YOLO-Seg instance segmentation model, performing pixel-level segmentation on the second target region, and outputting the mask contour of each target; generating a segmentation mask based on the mask contour.

[0012] In conjunction with the first aspect mentioned above, in one possible implementation, after outputting the classification results, the method further includes: summarizing the classification results of multiple tobacco images to generate detailed inventory data; automatically writing the detailed inventory data into the asset ledger system through a standardized data interface; and using an append-only log recording mechanism to store the write operation trajectory and modification records of the detailed inventory data.

[0013] Secondly, a tobacco brand classification device is provided, comprising: a communication unit and a processing unit; the communication unit is used to acquire multiple tobacco images; the processing unit is used to perform target detection on the multiple tobacco images based on a target detection model to obtain a first target region; match the first target region with a preset difficult-to-identify target library to determine a second target region; perform instance segmentation on the second target region based on an instance segmentation model to obtain a segmentation mask; perform deduplication counting based on the segmentation mask and obtain a target sub-image with background removed; perform fine-grained feature extraction on the regions of the first target region other than the second target region and the target sub-image based on a fine-grained feature extraction model to obtain a feature vector; and perform matching and retrieval of the feature vector with a standard vector library to output the classification result.

[0014] Thirdly, this application provides a tobacco brand sorting device, comprising: a processor and a storage medium; the storage medium includes instructions, and the processor is configured to execute the instructions to implement the method described in the first aspect and any possible implementation thereof. The tobacco brand sorting device may be an electronic device or a chip within an electronic device.

[0015] Fourthly, this application provides a tobacco brand classification system, including: an image acquisition module, a tobacco brand classification device, and a ledger data management module; wherein: the image acquisition module is used to acquire multiple tobacco display images under the same inventory task; the tobacco brand classification device is used to acquire multiple tobacco images; target detection is performed on the multiple tobacco images based on a target detection model to obtain a first target region; the first target region is matched with a preset difficult-to-identify target library to determine a second target region; instance segmentation is performed on the second target region based on an instance segmentation model to obtain a segmentation mask; deduplication and counting are performed based on the segmentation mask, and the data is obtained. The system first extracts the target sub-image by removing the background; then, based on a fine-grained feature extraction model, it extracts fine-grained features from the regions outside the second target region and the target sub-image within the first target region to obtain feature vectors; finally, it matches and retrieves the feature vectors against a standard vector library to output classification results; a ledger data management module is used to summarize the classification results of multiple tobacco images and generate detailed inventory data; the system automatically writes the detailed inventory data into the asset ledger system through a standardized data interface; and an append-only log recording mechanism is used to store the write operation trajectory and modification records of the detailed inventory data, ensuring that the inventory data is traceable and tamper-proof throughout the entire process.

[0016] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a tobacco brand sorting device, cause the tobacco brand sorting device to perform the methods described in the first aspect and any possible implementation thereof.

[0017] Sixthly, this application provides a computer program product containing instructions that, when run on a tobacco brand sorting device, cause the tobacco brand sorting device to perform the methods described in the first aspect and any possible implementation thereof.

[0018] This application provides a method and apparatus for classifying tobacco brands. It can acquire candidate regions through target detection and then divide these regions into regular and difficult-to-identify regions using a pre-defined difficult-to-identify target library. The difficult-to-identify regions are segmented to remove background and duplicate counts, resulting in a clean sub-image. Finally, fine-grained feature extraction and vector retrieval are performed on both the regular and clean sub-images to output the classification result. This scheme effectively overcomes the problems of low target counting and category recognition accuracy in densely stacked and partially occluded scenarios through a dual-path splitting and unified retrieval mechanism. It achieves high-precision, second-level automated identification of tobacco products, significantly reducing the cost and error rate of manual inventory checks.

[0019] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0020] Figure 1 This is a system architecture diagram of a tobacco brand classification system provided in Embodiment 1 of this application; Figure 2 A flowchart illustrating a tobacco brand classification method provided in Embodiment 2 of this application. Figure 1 ; Figure 3 A flowchart illustrating a tobacco brand classification method provided in Embodiment 3 of this application. Figure 2 ; Figure 4 A flowchart illustrating a tobacco brand classification method provided in Embodiment 4 of this application. Figure 3 ; Figure 5 A flowchart illustrating a tobacco brand classification method provided in Embodiment 5 of this application. Figure 4 ; Figure 6 A flowchart illustrating a tobacco brand classification method provided in Embodiment 6 of this application. Figure 5 ; Figure 7 A flowchart illustrating a tobacco brand classification method provided in Embodiment 7 of this application. Figure 6 ; Figure 8 A flowchart illustrating a tobacco brand classification method provided in Embodiment 8 of this application. Figure 7 ; Figure 9 This is a schematic diagram of the structure of a tobacco brand sorting device provided in Embodiment 11 of this application; Figure 10 This is a schematic diagram of the hardware structure of a tobacco brand classification device provided in Embodiment 11 of this application. Detailed Implementation

[0021] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0022] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0023] Example 1: The tobacco brand classification method provided in this application embodiment can be applied to, for example... Figure 1 In the tobacco brand classification system shown, such as Figure 1 As shown, the system includes: an image acquisition module 101, a tobacco brand classification device 102, and a ledger data management module 103.

[0024] The image acquisition module 101 is used to acquire multiple tobacco display images under the same inventory task.

[0025] The tobacco brand classification device 102 is used to acquire multiple tobacco images; perform target detection on the multiple tobacco images based on a target detection model to obtain a first target region; match the first target region with a preset difficult-to-identify target library to determine a second target region; perform instance segmentation on the second target region based on an instance segmentation model to obtain a segmentation mask; perform deduplication and counting based on the segmentation mask to obtain a target sub-image with background removed; perform fine-grained feature extraction on the regions of the first target region other than the second target region and the target sub-image based on a fine-grained feature extraction model to obtain a feature vector; match and retrieve the feature vector with a standard vector library to output the classification result.

[0026] The ledger data management module 103 is used to summarize the classification results of multiple tobacco images and generate detailed inventory data; it automatically writes the detailed inventory data into the asset ledger system through a standardized data interface; and it adopts an append-only log recording mechanism to store the writing operation trajectory and modification records of the detailed inventory data, ensuring that the inventory data is traceable and tamper-proof throughout the entire process.

[0027] Example 2: To address the technical problem of low accuracy in tobacco product recognition under scenarios of dense stacking and partial occlusion, this application provides a tobacco brand classification method, such as... Figure 2 As shown in the embodiments of this application, the tobacco brand classification method includes: S201. Acquire multiple tobacco images.

[0028] Among them, multiple tobacco images refer to multiple frames of RGB images of tobacco display collected under the same inventory task, covering the visual data of tobacco packs.

[0029] In this embodiment, the tobacco brand classification device receives multiple tobacco images uploaded by an external acquisition device through a communication unit, and binds them uniformly to the corresponding inventory task number to complete the batch access and task association of image data.

[0030] It should be noted that image acquisition supports multiple methods such as real-time shooting and uploading, selection from local albums, and retrieval of historical records, adapting to different inventory operation scenarios.

[0031] As an example, workers use their mobile devices to take pictures of cigarettes displayed on the shelves, uploading up to nine high-definition images at a time. The device then receives these images and binds them to the store's inventory task number for the day.

[0032] Based on the above steps, unified access and task-oriented management of multi-source tobacco images were achieved, providing complete raw image data for subsequent batch recognition and processing.

[0033] S202. Target detection is performed on multiple tobacco images based on the target detection model to obtain the first target region.

[0034] The first target region refers to the region of interest (ROI) sub-map containing a single tobacco product unit after target detection, localization, and cropping. It is the basic unit for subsequent diversion and recognition processing.

[0035] In this embodiment, the tobacco brand classification device inputs multiple tobacco images into the target detection model, and through high-resolution adaptation and parameter optimization processing, outputs the bounding boxes and detection confidence scores of all tobacco products. The ROI sub-image of each tobacco product is obtained by cutting the bounding boxes one by one, which serves as the first target region.

[0036] It should be noted that the first target area has only been roughly located and cropped, and the area may still contain overlapping parts of background pixels and adjacent goods, which require further processing.

[0037] Based on the above steps, the rapid localization and cropping of tobacco units in the entire scene was completed. Even in densely stacked scenes, the actual boundaries of tobacco products can still be preserved, providing accurate candidate regions for subsequent diversion processing.

[0038] S203. Match the first target region with a preset database of difficult-to-identify targets to determine the second target region.

[0039] The preset difficult-to-identify target library is a collection of labels for categories with high historical identification errors and categories with highly similar appearances. The second target area is the tobacco product area that hits the difficult-to-identify label and requires refined processing.

[0040] In this embodiment, the tobacco brand classification device compares the coarse classification result of the first target region with the difficult-to-identify category labels in the database. If a match is found, it indicates that directly extracting features from the target region is prone to misjudgment and requires more refined preprocessing. In this case, the first target region corresponding to the matched label is determined as the second target region, and the remaining regions are designated as regular, easily identifiable regions.

[0041] As an example, the coarse classification result of a certain first target area is two slim cigarettes with similar appearances. They match the corresponding labels in the preset difficult-to-identify target library. The device identifies them as the second target area and transfers them to the instance segmentation processing path.

[0042] Based on the above steps, a logical starting point for dual-path splitting is established, which distinguishes between easily identifiable targets and difficult-to-identify targets, avoiding high-computational-consumption instance segmentation processing for all targets, thereby significantly improving overall processing efficiency while ensuring recognition accuracy.

[0043] S204. Perform instance segmentation on the second target region based on the instance segmentation model to obtain the segmentation mask.

[0044] The segmentation mask is a binary image that identifies the pixel-level boundaries of the cigarette product. Foreground pixels correspond to the cigarette product area, and background pixels correspond to the non-cigarette product area. It can accurately depict the outline of overlapping and occluded cigarette products. An instance refers to a single independent cigarette product unit, and the mask outline is a set of pixels on the edge of the cigarette product.

[0045] In this embodiment, the tobacco brand classification device inputs the second target region into the instance segmentation model, performs pixel-level segmentation on the tobacco products within the region, outputs the mask contour of each independent tobacco product, and generates a binarized segmentation mask based on the mask contour.

[0046] It should be noted that instance segmentation is only performed on the second target region; regular regions are not included in this processing step to avoid unnecessary consumption of computing power.

[0047] Based on the above steps, pixel-level accurate boundary extraction of cigarette products in difficult-to-identify areas was achieved, solving the problem that the bounding box could not accurately depict the outline of cigarette products in overlapping and occluded scenes.

[0048] S205. Perform deduplication counting based on the segmentation mask and obtain the target sub-image with the background removed.

[0049] Among them, the target sub-image is a pure single-item image that removes background interference and contains only the texture information of a single cigarette product, and the deduplication counting is the process of counting the actual number of cigarette products after eliminating duplicate masks.

[0050] In this embodiment, the tobacco brand classification device calculates the minimum bounding rectangle and aspect ratio features of each tobacco product based on the segmentation mask, and combines non-maximum suppression (NMS) to filter overlapping duplicate masks. The tobacco product count is completed based on the number of duplicate masks. At the same time, a guide clipping box is generated based on the mask outline, and the second target area is clipped to obtain the target sub-image with the background removed.

[0051] It should be noted that the target sub-image only retains the foreground pixels within the mask range, while the background pixels are uniformly set to invalid values ​​to eliminate external interference to the greatest extent.

[0052] Based on the above steps, the counting of cigarettes and background removal in densely stacked scenarios were accurately completed, which not only ensured the accuracy of the inventory count but also improved the image purity of subsequent feature extraction.

[0053] S206. Based on the fine-grained feature extraction model, fine-grained feature extraction is performed on the regions other than the second target region and the target sub-image in the first target region to obtain feature vectors.

[0054] Among them, the feature vector is a high-dimensional vector that represents the visual semantic features of tobacco products. Fine-grained feature extraction can capture the subtle textures and pattern differences of tobacco products, which is the core basis for category matching.

[0055] In this embodiment, the tobacco brand classification device inputs the conventional easily identifiable region and the target sub-image after background removal into the fine-grained feature extraction model to extract the original feature vector. After normalizing the original feature vector, the features corresponding to invalid ROIs are filtered out, and the effective high-dimensional feature vectors are retained.

[0056] It should be noted that the images from the two divergent paths are uniformly entered into the feature extraction stage here, and the output feature vectors are in the same feature space, ensuring the consistency of subsequent retrieval logic.

[0057] Based on the above steps, unified feature extraction of dual-path output images was achieved, eliminating the interference of illumination and background differences on features, and providing highly discriminative feature data for subsequent vector retrieval.

[0058] S207. Match and retrieve the feature vectors with the standard vector library, and output the classification results.

[0059] The standard vector library is a database that stores standard feature vectors for various brands and specifications of cigarettes. The matching retrieval is a process of matching corresponding categories through similarity calculation. The classification results include brand, specification and quantity information.

[0060] In this embodiment of the application, the tobacco brand classification device calculates the similarity between the feature vector and each vector in the standard vector library, outputs the corresponding classification result based on the comparison result of the similarity with the preset threshold, and determines the final result after auxiliary disambiguation of multiple candidate results with close similarity by combining aspect ratio features.

[0061] It should be noted that the auxiliary disambiguation is only triggered when the difference in similarity between multiple candidate results is less than the threshold. In normal scenarios, the classification result corresponding to the highest similarity is directly output, balancing accuracy and efficiency.

[0062] Based on the above steps, fine-grained category matching of tobacco products was completed. Combined with geometric feature-assisted disambiguation, the problem of confusion between similar specifications was solved, and high-precision brand classification results were output.

[0063] Based on the above technical solution, the tobacco brand classification device effectively solves the problem of low tobacco product recognition accuracy in densely stacked and partially occluded scenarios through a full-process processing of target detection and candidate region positioning, dual-path diversion of difficult-to-identify target library, instance segmentation for deduplication and background removal, and unified fine-grained feature extraction and vector retrieval. It realizes automated and high-precision recognition of tobacco brands and quantities, and greatly improves the efficiency and accuracy of inventory counting.

[0064] Example 3: In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S202 can be specifically implemented through the following S301, S302 and S303, which are explained in detail below: S301. Input multiple tobacco images into the YOLOv11 target detection model.

[0065] Among them, the YOLOv11 object detection model is a single-stage deep learning detection network that can simultaneously output object bounding boxes and coarse classification confidence scores.

[0066] In this embodiment, the tobacco brand classification device performs size normalization and channel verification on multiple tobacco images under the same inventory task, and then inputs them in batches into the pre-trained YOLOv11 target detection model.

[0067] It should be noted that the input images must be in RGB three-channel format to avoid inference dimension mismatch anomalies caused by grayscale images or images with transparent channels. The YOLOv11 object detection model uses an improved CSPDarknet as the backbone network to extract multi-scale features from the input images. This backbone network, through a cross-stage partial connection structure, reduces computational redundancy while retaining rich gradient flow information, enhancing its ability to learn fine-grained textures and patterns on cigarette packaging. After the backbone network, the model uses a path aggregation network (PANet) as the neck structure, which efficiently integrates deep semantic information with shallow detail information through a bidirectional feature fusion path from top to bottom and bottom to top, thereby improving the robustness of multi-scale cigarette target detection in densely stacked and partially occluded scenes. The model's prediction head adopts a decoupled detection head structure, outputting the bounding box regression parameters, target confidence, and coarse classification probability of the target, thereby achieving end-to-end target localization and preliminary category prediction.

[0068] As an example, the device uniformly scales up nine images of shelf displays uploaded by stores to the appropriate size, then inputs them in batches into the YOLOv11 model deployed in the cloud inference cluster, and starts forward computation.

[0069] Based on the above steps, a standardized model input for multi-source tobacco images was completed, ensuring the consistency and stability of subsequent detection inference.

[0070] S302. Multiple tobacco images are processed by high-resolution adaptation and non-maximum suppression (NMS) parameter adjustment, and the bounding boxes and detection confidence scores of all tobacco products are output.

[0071] Among them, high-resolution adaptation refers to increasing the pixel size of the input image to preserve the fine-grained texture of the tobacco product, and non-maximum suppression (NMS) is used to filter redundant bounding boxes that overlap repeatedly.

[0072] In this embodiment, the tobacco brand classification device adapts the input image resolution to 1536×1536, lowers the NMS intersection-over-union threshold to 0.3, raises the detection confidence threshold, and outputs the bounding box coordinates and classification confidence value of each tobacco product after model inference.

[0073] It should be noted that the NMS threshold can be dynamically adjusted according to the density of the scene. In warehouse stacking scenarios, the threshold can be further lowered to avoid missing closely packed tobacco products.

[0074] As an example, for a scenario with densely arranged slim cigarette packs, the device sets the NMS crossover ratio threshold to 0.28 and the confidence threshold to 0.6. After inference, it outputs the coordinates of the top left and bottom right corners of all cigarette packs in the image and their corresponding confidence scores.

[0075] Based on the above steps, the bounding boxes of real cigarette products are preserved in densely stacked and partially occluded scenarios, effectively reducing the probability of missed detection and false deletion of adjacent targets.

[0076] S303. Obtain the Region of Interest (ROI) sub-map for each tobacco product based on bounding box clipping, and use it as the first target region.

[0077] The region of interest (ROI) is a local image unit cropped from the original image that contains a single tobacco product target.

[0078] In this embodiment of the application, the tobacco brand classification device performs pixel cropping at the corresponding position of the original tobacco image based on the coordinate information of each output bounding box, generates ROI sub-images that correspond one-to-one with the tobacco products, and uniformly collects them into the first target area.

[0079] It should be noted that when cropping, a margin of 2 to 5 pixels can be retained outside the bounding box to avoid cutting off the texture and marking information of the cigarette product edges due to the positioning deviation of the bounding box.

[0080] As an example, the device outputs 24 bounding boxes from an image, and then sequentially crops 24 independent ROI sub-images of cigarette packs from the original image, sorts them according to detection confidence, and stores them in the first target region processing queue.

[0081] Based on the above steps, the independent splitting of the tobacco product unit in the whole image was completed, transforming the whole image recognition into a single target processing unit, which improved the flexibility of subsequent diversion and classification processing.

[0082] Based on the above technical solution, the YOLOv11 model is adapted specifically for tobacco scenarios through high-resolution adaptation and optimized NMS parameters. In densely stacked display scenarios, the accurate positioning and cropping of tobacco targets are achieved, which not only ensures the detection rate of dense targets, but also provides accurate candidate regions for subsequent diversion of difficult-to-identify targets and fine-grained identification.

[0083] Example 4: In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S203 can be specifically implemented through the following S401, S402 and S403, which are explained in detail below: S401. Obtain the coarse classification result of the first target region.

[0084] The coarse classification result is the preliminary category prediction result output synchronously by the target detection model, corresponding to the preliminary determination label of tobacco brand and specification.

[0085] In this embodiment, the tobacco brand classification device reads the predicted probability value of the category corresponding to each first target region output by the target detection model, and takes the category with the highest probability as the coarse classification result of that region.

[0086] It should be noted that the coarse classification results are only used for traffic division and are not directly used as the final classification output. The accuracy requirement is lower than that of the final fine-grained recognition results.

[0087] Based on the above steps, the preliminary category attributes of each target area were quickly obtained, providing a basis for subsequent matching and triage of difficult-to-identify targets.

[0088] S402. Match the coarse classification results with the difficult-to-identify category labels in the preset difficult-to-identify target library.

[0089] Among them, the preset difficult-to-identify target library is a pre-built collection that stores historically high-error categories and highly similar-looking category labels.

[0090] In this embodiment, the tobacco brand classification device traverses all tags in a preset difficult-to-identify target library, performs string matching and semantic comparison between the current coarse classification result and each tag, and outputs the matching hit status.

[0091] Optionally, the tag matching similarity satisfies the following formula:

[0092] in, The similarity between the coarse classification result labels and the hard-to-identify labels. The edit distance between the two label texts. , These represent the character lengths of the two label texts, respectively.

[0093] It should be noted that the preset difficult-to-identify target library supports dynamic updates, and difficult-to-identify category labels can be added or adjusted periodically based on actual identification error data.

[0094] As an example, the device compares the coarse classification result "a certain brand of slim soft box" with the labels in the difficult-to-identify target library one by one, confirms that the label has been included in the library, and returns the judgment result of matching.

[0095] Based on the above steps, automated matching and verification of coarse classification results and difficult-to-identify labels were achieved, providing accurate trigger judgment basis for dual-path traffic splitting.

[0096] S403. If the coarse classification result hits any difficult-to-identify category label in the preset difficult-to-identify target library, then the corresponding first target area is determined as the second target area.

[0097] The second target area is the tobacco product area that has a difficult-to-identify label and needs to be processed in the instance segmentation path.

[0098] In this embodiment, the tobacco brand classification device marks the first target area that matches the difficult-to-identify label as the second target area and includes it in the instance segmentation processing queue, based on the matching results. The areas that do not match are included in the regular identification queue.

[0099] It should be noted that if the first target region of the difficult-to-identify target library is not hit, it does not need to enter the instance segmentation stage, but will directly proceed to the subsequent fine-grained feature extraction step.

[0100] As an example, in a batch of 32 first target regions, 6 of them were coarsely classified and matched the difficult-to-identify labels. The device identified these 6 regions as second target regions, and the remaining 26 regions directly entered the subsequent feature extraction process.

[0101] Based on the above steps, a dual-path split was achieved between conventional targets and difficult-to-identify targets, avoiding redundant high-computation processing for easily identifiable targets.

[0102] Based on the above technical solution, a dual-path intelligent diversion of the tobacco identification process is realized through the matching mechanism between coarse classification results and a preset difficult-to-identify target library. A refined processing path is initiated for difficult-to-identify categories, while an efficient identification path is maintained for regular categories. While ensuring the identification accuracy in complex scenarios, the overall computing power consumption and processing latency are effectively controlled.

[0103] Example 5: In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 5 As shown, the above S204 can be implemented through the following S501 and S502, which are explained in detail below: S501. Input the second target region into the YOLO-Seg instance segmentation model, perform pixel-level segmentation on the second target region, and output the mask contour of each target.

[0104] Among them, the YOLO-Seg instance segmentation model is an end-to-end single-stage instance segmentation network, and the mask contour is a set of pixels that identify the edge of the cigarette product.

[0105] In this embodiment, the tobacco brand classification device inputs the second target region obtained by diversion into the pre-trained YOLO-Seg instance segmentation model in batches. The model performs pixel-level foreground and background classification for each tobacco product in the region and outputs the set of mask contour coordinate points corresponding to each independent tobacco product.

[0106] It should be noted that the YOLO-Seg instance segmentation model uses CSPDarknet, similar to YOLOv11, as a shared backbone network to extract deep visual features of the second target region. Following the backbone network, the model's feature pyramid network fuses multi-scale features to enhance adaptability to different sizes of tobacco products (such as cigarette packs and cartons). Unlike simple object detection models, the YOLO-Seg model connects both the detection head and the segmentation head after the feature fusion network. The segmentation head, composed of convolutional layers, is responsible for mapping each pixel on the fused feature map to the target category and generating a set of prototype mask coefficients for each detected instance. Finally, by linearly combining the prototype mask with the predicted mask coefficients, a pixel-level binary mask contour for each tobacco product instance is obtained, thus achieving fine-grained segmentation of the edges of individual tobacco product units in densely stacked scenes.

[0107] As an example, the device inputs six tobacco product regions with hard-to-identify labels into the YOLO-Seg model, and outputs the edge pixel coordinates of each tobacco product in each region after inference, resulting in a total of 11 independent mask contour point sets.

[0108] Based on the above steps, the precise pixel-level boundaries of each cigarette product in the difficult-to-identify area were obtained, solving the problem that traditional bounding boxes cannot depict the true outline of overlapping and occluded cigarette products.

[0109] S502. Generate a segmentation mask based on the mask contour.

[0110] The segmentation mask is a binary image, where foreground pixels correspond to the tobacco product area and background pixels correspond to the non-tobacco product area.

[0111] In this embodiment, the tobacco brand classification device fills the foreground pixel values ​​on a blank canvas with the same size as the second target area according to the coordinate point set of each mask outline, thereby generating a single-channel binarized segmentation mask for the corresponding tobacco product.

[0112] It should be noted that the segmentation mask corresponds one-to-one with the pixel coordinates of the original second target region, and can be directly used for subsequent image cropping and pixel filtering operations.

[0113] As an example, the device generates 11 single-channel binary images sequentially based on the 11 sets of output mask contours, with the foreground pixel value set to 255 and the background pixel value set to 0, as the segmentation mask for the corresponding tobacco product.

[0114] Based on the above steps, a standardized segmentation mask that can be directly used for subsequent processing is obtained, providing accurate pixel-level basis for deduplication counting and background removal.

[0115] Based on the above technical solution, the YOLO-Seg instance segmentation model is used to perform pixel-level fine segmentation of difficult-to-identify regions, accurately extracting the independent segmentation mask of each cigarette product in densely stacked and partially occluded scenes. This breaks through the accuracy limit of traditional bounding box localization and provides reliable pixel-level data support for subsequent accurate counting and clean target image extraction.

[0116] Example 6: In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 6 As shown, the above S205 can be implemented through the following S601 to S606, which are explained in detail below: S601. Calculate the minimum bounding rectangle of the corresponding target based on the segmentation mask.

[0117] The minimum bounding rectangle refers to the smallest axis-aligned rectangle that can completely enclose all foreground pixels of a single segmentation mask.

[0118] In this embodiment of the application, the tobacco brand classification device traverses all foreground pixels within a single segmentation mask, extracts the extreme coordinates in the horizontal and vertical directions respectively, and generates the minimum bounding rectangle of the corresponding target based on the extreme coordinates.

[0119] It should be noted that the minimum bounding rectangle is calculated using an axis-aligned method, which eliminates the need to consider the target's rotation angle, ensuring computational efficiency and consistency with subsequent geometric features.

[0120] As an example, the device performs pixel traversal on the segmentation mask of a cigarette pack, obtaining the minimum horizontal coordinate value of 12 and the maximum horizontal coordinate value of 214, and the minimum vertical coordinate value of 36 and the maximum vertical coordinate value of 198, and generates the minimum bounding rectangle corresponding to the mask accordingly.

[0121] Based on the above steps, the rectangular boundary parameters of each tobacco mask were obtained, providing a geometric reference for subsequent aspect ratio calculation and mask deduplication.

[0122] S602. Calculate the aspect ratio feature based on the minimum bounding rectangle.

[0123] The aspect ratio is the ratio of the length of the longer side to the length of the shorter side of the smallest bounding rectangle, and is used to characterize the geometric properties of the shape of tobacco products.

[0124] In this embodiment, the tobacco brand classification device reads the width and height pixel values ​​of the smallest bounding rectangle, divides the value of the longer side by the value of the shorter side to obtain the aspect ratio feature value of the corresponding target, and binds and stores it with the mask.

[0125] It should be noted that the aspect ratio is strongly correlated with the physical specifications of the cigarette products. Each type of cigarette pack, hard pack, and soft pack has a stable aspect ratio range, which can be used as an auxiliary basis for subsequent disambiguation.

[0126] As an example, the device calculates that the minimum bounding rectangle of a certain smoke is 420 pixels wide and 120 pixels high, and calculates the aspect ratio to be 3.5, which is then bound and stored as the geometric feature of the target.

[0127] Based on the above steps, the inherent geometric features of tobacco products were extracted, providing a quantifiable auxiliary judgment dimension for subsequent mask deduplication and category disambiguation.

[0128] S603. Combine the number of mask instances with Non-Maximum Suppression (NMS) to perform deduplication on overlapping masks.

[0129] The number of mask instances is the total number of initially generated segmentation masks within the current second target region, and NMS is used to filter duplicate overlapping masks corresponding to the same target.

[0130] In this embodiment, the tobacco brand classification device first counts the number of initial mask instances in the current area, and then performs the NMS algorithm based on the pixel-level intersection-union ratio and mask confidence to eliminate overlapping masks that are predicted repeatedly and retain effective masks with higher confidence.

[0131] Optionally, the pixel-level intersection-union ratio of mask deduplication satisfies the following formula:

[0132] in, Let the pixel-level intersection-union ratio of the two segmentation masks be . This represents the number of intersections between the foreground pixels of the two masks. The number of pixels in the union of the foreground pixels of the two masks.

[0133] It should be noted that the crossover ratio threshold of NMS can be adjusted according to the scenario. The more densely stacked the scene, the lower the threshold should be set to avoid misjudging adjacent independent smoke products as duplicate masks.

[0134] As an example, 11 mask instances are initially generated for a certain second target region, and the device is set to... With a threshold of 0.6, NMS is performed to remove 3 overlapping duplicate masks, ultimately retaining 8 valid masks.

[0135] Based on the above steps, redundant masks generated by repeated predictions in dense scenes are eliminated, ensuring that each real cigarette product corresponds to a unique and effective segmentation mask.

[0136] S604. Target counting is completed based on the number of deduplicated mask instances.

[0137] The number of duplicate mask instances is the total number of valid masks retained after NMS filtering, which corresponds one-to-one with the actual number of cigarette products.

[0138] In this embodiment, the tobacco brand classification device counts the total number of valid masks retained after deduplication, uses this value as the actual quantity of tobacco products in the current second target area, and writes it into the inventory count result.

[0139] It should be noted that the counting results are counted separately according to the tobacco product category, and the masks of different coarse classification labels are counted separately to avoid confusion in the number of different categories.

[0140] As an example, after deduplication of a certain second target area, 8 valid masks are retained, of which 5 correspond to brand A slim cigarettes and 3 correspond to brand B hard-pack cigarettes. The device counts the corresponding categories respectively.

[0141] Based on the above steps, the true number of cigarettes in densely stacked scenarios was accurately obtained, solving the problem of large counting deviation in traditional bounding box detection in overlapping scenarios.

[0142] S605. Contour generation guides the clipping frame based on segmentation mask.

[0143] Among them, the guided clipping box is the clipping boundary generated by fitting the outer contour of the segmentation mask, which is used to accurately select the effective pixel area of ​​the target tobacco product.

[0144] In this embodiment, the tobacco brand classification device performs contour extraction on the deduplicated effective segmentation mask to obtain the outer contour point set of the mask, and generates a closely fitting clipping frame based on the outer contour point set as a guide clipping frame.

[0145] It should be noted that the guide cropping box can reserve a buffer margin of 1 to 3 pixels around the outline to avoid the loss of detail information of edge texture due to cropping.

[0146] As an example, the device uses a contour extraction algorithm to obtain the outer contour point set of a mask, fits and generates a rectangular clipping box that tightly surrounds the contour, and extends it outward by 2 pixels as the final guiding clipping box.

[0147] Based on the above steps, a cropping benchmark that accurately matches the actual boundaries of the tobacco product is generated, avoiding the problem of traditional rectangular cropping introducing a large number of background pixels.

[0148] S606. Use the guided cropping box to crop the second target area, remove background pixels, and use the resulting clean single-item image as the target sub-image.

[0149] The target sub-image is a clean single-item image that has had background interference removed and only retains the foreground texture of a single tobacco product.

[0150] In this embodiment, the tobacco brand classification device performs cropping in the original image of the second target area according to the coordinates of the guided cropping box, and at the same time, combines the segmentation mask to set the non-foreground pixels in the cropping area to invalid values, finally obtaining a pure sub-image containing only the target tobacco products.

[0151] It should be noted that the background pixels are uniformly set to pure black or transparent values ​​to avoid residual background textures interfering with subsequent feature extraction.

[0152] As an example, after the device cuts out the tobacco product area according to the guided clipping frame, it sets all background pixels outside the mask to 0, resulting in a clean image that retains only the cigarette pack pattern and logo, which is then stored in the processing queue as the target sub-image.

[0153] Based on the above steps, a clean tobacco image without background interference was obtained, which significantly increased the proportion of effective information in subsequent fine-grained feature extraction.

[0154] Based on the above technical solution, accurate cigarette counting in densely stacked scenarios is achieved through geometric calculation of segmentation masks and NMS deduplication. At the same time, background pixels are completely removed by relying on mask contour-guided cropping. This not only ensures the accuracy of inventory counts but also outputs high-purity single-item images, effectively reducing the negative impact of background and overlapping interference on subsequent recognition accuracy.

[0155] Example 7: In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 7 As shown, the above S206 can be implemented through the following S701, S702 and S703, which are explained in detail below: S701. Use the DINO model to extract features from the regions and target sub-images in the first target region, excluding the second target region.

[0156] Among them, DINO is a pre-trained visual Transformer model based on self-supervised learning, which can extract high-dimensional fine-grained semantic features of images.

[0157] In this embodiment, the tobacco brand classification device inputs the conventional easily identifiable region and the target sub-image after background removal into the DINO pre-trained model. The model encodes the local texture and global structure information of the image through the multi-head self-attention mechanism and outputs the original high-dimensional feature vector.

[0158] It should be noted that the DINO model employs a Vision Transformer (ViT)-based backbone network. The core of this backbone network lies in its multi-head self-attention layer, which captures long-distance dependencies between different regions in an image (e.g., trademarks, warnings, and color blocks on tobacco packaging). Specifically, the input target sub-image or regular ROI region is divided into a series of image patches and converted into sequential embedding vectors through linear mapping. The DINO model encodes these embedding vectors using stacked Transformer encoder layers and their self-attention mechanism, thereby generating a high-dimensional global feature vector. This feature vector implicitly encodes the global layout and local semantic details of the tobacco image. In this approach, the DINO model uses pre-trained weights based on self-supervised learning, eliminating the need for additional training specific to tobacco categories. This allows it to extract general visual features robust to changes in illumination and minor deformations, providing highly discriminative feature representations for subsequent vector matching retrieval.

[0159] As an example, the device batch inputs 26 regular ROI sub-images and 8 background-removed target sub-images into the DINO model. After inference, each outputs a 768-dimensional original feature vector, which is then uniformly stored in the feature cache queue.

[0160] Based on the above steps, unified fine-grained feature extraction of dual-path images was completed, and visual semantic representations of tobacco products with high discriminative power were obtained.

[0161] S702. Perform L2 normalization on the extracted original feature vectors.

[0162] L2 normalization is a mathematical process that scales vectors to a unit length, eliminating the impact of differences in the overall magnitude of vectors on similarity calculation.

[0163] In this embodiment, the tobacco brand classification device calculates the L2 norm of each original feature vector, divides each element in the vector by the norm value, and obtains a normalized feature vector with a modulus of 1.

[0164] It should be noted that the similarity calculation of normalized vectors is determined only by the vector direction, which can reduce the interference of feature amplitude fluctuations caused by differences in illumination and contrast.

[0165] As an example, the device calculates the L2 norm of a 768-dimensional original feature vector as 12.6, divides the value of each dimension by 12.6, and obtains a normalized feature vector with a modulus of 1.

[0166] Based on the above steps, the units and magnitudes of the feature vectors are unified, which improves the stability and robustness of subsequent vector similarity matching.

[0167] S703: Filter out invalid Region of Interest (ROI) features that are too small or completely black, and retain the effective high-dimensional feature vectors as feature vectors.

[0168] Invalid ROI features refer to feature vectors whose corresponding image size is insufficient or whose pixel information is missing, thus failing to support effective category identification.

[0169] In this embodiment, the tobacco brand classification device determines the ROI based on the pixel area threshold and the average pixel value threshold, and removes the feature vectors corresponding to ROIs with too small an area or an average pixel value close to pure black, and retains the remaining ones as the final effective feature vectors.

[0170] It should be noted that invalid ROIs are mostly caused by severe occlusion and edge truncation, and their feature noise accounts for a high proportion. Retaining them will interfere with the accuracy of subsequent retrieval and matching.

[0171] As an example, the device sets the area threshold to 32×32 pixels and the pixel mean threshold to 10. If a certain ROI has a size of 20×18 pixels and a pixel mean of 5, it is determined to be an invalid ROI, and its corresponding feature vector is filtered out.

[0172] Based on the above steps, the feature vector set was purified, the interference of low-quality invalid samples was eliminated, and the accuracy of subsequent retrieval and matching was ensured.

[0173] Based on the above technical solution, fine-grained features of dual-path images are extracted uniformly through the DINO model, and L2 normalization is combined to improve feature robustness. Invalid ROI features are filtered out to remove noise interference. While ensuring feature discriminability, the unification of the dual-path feature space is achieved, providing high-quality feature data support for subsequent vector retrieval and category matching.

[0174] Example 8: In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 8 As shown, the above S207 can be implemented through the following S801 and S802, which are explained in detail below: S801. Calculate the similarity between the feature vector and each standard vector in the standard vector library; if the similarity is greater than or equal to the preset threshold, output the corresponding brand or specification classification result; if the similarity is less than the preset threshold, output the unknown category result.

[0175] The standard vector library is a database that pre-stores standard feature vectors for various brands and specifications of cigarettes. The similarity is used to measure the degree of semantic matching between the feature to be identified and the standard feature.

[0176] In this embodiment, the tobacco brand classification device retrieves all L2-normalized standard feature vectors from the standard vector library, calculates the similarity of each feature vector to be identified, and compares the highest similarity result with a preset threshold. If the threshold condition is met, the corresponding brand or specification is output; otherwise, if all are below the threshold, an unknown category result is output.

[0177] Preferably, the similarity is calculated by calculating the cosine similarity of the vectors.

[0178] It should be noted that since the feature vectors have all undergone L2 normalization and the vector magnitude is 1, the cosine similarity calculation can be simplified to a vector dot product operation, which can effectively improve the computational efficiency of vector retrieval.

[0179] Based on the above steps, a rapid matching and determination of tobacco product features with the standard vector library was completed, realizing the automated identification and output of tobacco product categories in normal scenarios.

[0180] S802. For multiple candidate classification results with similarity differences less than a preset difference threshold, auxiliary disambiguation is performed by combining the aspect ratio features corresponding to the segmentation mask, and the final classification result is determined from the multiple candidate classification results.

[0181] Among them, the preset difference threshold is the critical value for determining the similarity of multiple candidate results, and the auxiliary disambiguation is a processing mechanism that introduces geometric features to supplement the distinction of similar categories.

[0182] In this embodiment of the application, the tobacco brand classification device calculates the similarity difference between the top-ranked candidate classification results. If the difference is less than a preset difference threshold, the aspect ratio feature of the corresponding target is retrieved and compared with the standard aspect ratio of each candidate category. The candidate with the highest matching degree is selected as the final classification result.

[0183] Optionally, the aspect ratio matching degree satisfies the following formula: in, For aspect ratio matching, The aspect ratio of the target to be identified. The standard aspect ratio corresponding to the candidate product category.

[0184] It should be noted that the auxiliary disambiguation is only triggered when the similarity of multiple candidate results is close. This logic is not executed in the usual single-optimal-result scenario, thus balancing recognition accuracy and overall processing efficiency.

[0185] As an example, the similarity between a target to be identified and slim cigarettes of brand B and brand C is 0.82 and 0.80, respectively. The difference of 0.02 is less than the preset difference threshold of 0.05. The device retrieves its aspect ratio of 3.2 and compares it with the standard aspect ratios of 3.15 and 2.9 for the two categories, respectively. The slim cigarette of brand B, which has a higher matching degree, is determined as the final classification result.

[0186] Based on the above steps, the problem of confusion in pure feature retrieval for similar-looking categories is solved, and the recognition accuracy of fine-grained similar specifications is effectively improved.

[0187] Based on the above technical solution, rapid classification of tobacco categories is achieved through vector similarity matching. At the same time, an auxiliary disambiguation mechanism is constructed by combining aspect ratio geometric features. While ensuring recognition efficiency in normal scenarios, the problem of confusion and misjudgment of cigarettes with similar packaging specifications is effectively solved, and the accuracy and scenario robustness of tobacco brand classification are comprehensively improved.

[0188] Example 9: This embodiment elaborates on the inventory data flow and ledger recording mechanism.

[0189] After outputting the classification results, the system aggregates the classification results from multiple tobacco images to generate detailed inventory data. Specifically, once fine-grained feature extraction and vector retrieval have been completed for multiple tobacco images bound under the same inventory task, the system aggregates and statistically analyzes the structured classification results of all images, including brand, specifications, quantity, and placement coordinates. For example, the system groups and aggregates the recognition results according to dimensions such as store, manufacturer, cigarette brand, and specifications, automatically calculates the shelf placement rate and product proportion of each type of cigarette, and removes fuzzy, duplicate, or low-confidence invalid recognition targets, ultimately generating detailed inventory data for the inventory task.

[0190] After generating detailed inventory count data, the data is automatically written to the asset ledger system via a standardized data interface. Specifically, the standardized data interface can be a REST API or a GraphQL interface. The system internally encapsulates the inventory count details and automatically sends a write request to the asset ledger system's interface, eliminating the need for manual data entry. For example, the interface request message may include store identifier, operator information, data length, and business content fields. After receiving and verifying permissions, the asset ledger system persistently stores the inventory count details in a relational database or distributed storage system. This automated data flow mechanism eliminates manual data processing and entry, effectively reducing the probability of human error and achieving an end-to-end closed loop from image acquisition and recognition to data storage.

[0191] To ensure the security and compliance of ledger data, an append-only logging mechanism is adopted to store the write operation trajectory and modification records of inventory count details. Specifically, the append-only logging mechanism is a data storage method that only allows appending writes and does not permit modification or deletion of historical records. Its technical implementation logic is similar to the chain structure of a blockchain: each time a ledger write or modification operation is performed, the system generates a log record containing an operation timestamp, operator identification, operation type (e.g., new inventory count, quantity correction), original data snapshot, and modified data snapshot. Subsequently, the hash value of this log record is calculated, and the hash value of the previous log record is embedded into the current record, forming a hash chain. Since any tampering with historical logs will result in a hash value mismatch, which will be immediately detected by the system, this mechanism guarantees the immutability of data from the underlying algorithm. This embodiment, by adopting an append-only logging mechanism, completely retains the trajectory of every inventory count, data modification, and inbound / outbound operation, achieving full traceability of inventory records and modification operations, and meeting the stringent compliance requirements of asset auditing in the tobacco industry.

[0192] Example 10: To more clearly illustrate the tobacco brand classification method provided in this application, the following example demonstrates the full-process application of this method in a real-world environment, using a store's mobile device initiating an inventory check from the cloud. This embodiment showcases the effectiveness of the technical solution in complex retail scenarios, significantly improving business efficiency by transitioning from manual inventory checks to automated identification.

[0193] In this application scenario, staff perform inventory tasks using mobile devices in the store. First, the mobile device acquires multiple images of tobacco products. Specifically, staff can use the mobile device's camera to take photos of the cigarettes displayed on the shelves, or select historical images in batches from their local photo album. The mobile device supports uploading multiple high-resolution images concurrently, for example, selecting up to nine images no larger than 3MB at a time. Through segmented upload and breakpoint resume mechanisms, the image data stream is stably transmitted to the tobacco brand classification device even in weak network environments. Upon receiving the batch of images, the tobacco brand classification device binds them all to the same inventory task number for subsequent batch parsing, processing, and data aggregation.

[0194] After acquiring multiple tobacco images, the tobacco brand classification device performs object detection on these images using an object detection model to obtain the first target region. Specifically, the tobacco brand classification device is equipped with a high-performance graphics processing unit cluster, which inputs the received multiple tobacco images into the YOLOv11 object detection model. Since cigarettes on store shelves are often densely stacked, the model processes the multiple tobacco images through high-resolution adaptation and non-maximum suppression (NMS) parameter adjustment, outputting bounding boxes and detection confidence scores for all tobacco products. Based on the bounding boxes, a region of interest (ROI) sub-image is obtained for each tobacco product, serving as the first target region. In this stage, the tobacco brand classification device quickly scans the entire image, locating all visible cigarette packs, cartons, or boxes in the picture.

[0195] After obtaining the first target area, the tobacco brand classification device matches it with a preset difficult-to-identify target library to determine the second target area. Specifically, the tobacco brand classification device obtains the coarse classification result of the first target area and matches it with the difficult-to-identify category labels in the preset difficult-to-identify target library. If the coarse classification result matches any difficult-to-identify category label in the preset difficult-to-identify target library, the corresponding first target area is determined as the second target area. For example, some cigarettes of different specifications with highly similar appearances are easily confused in the coarse classification stage. The system accurately filters out these targets that require fine processing by querying the dynamically constructed difficult-to-identify category label library, thereby establishing a logical starting point for dual-path diversion, avoiding redundant processing of regular easily identifiable targets, and significantly improving overall processing efficiency while ensuring recognition accuracy.

[0196] For the selected second target region, the tobacco brand classification device performs instance segmentation on the second target region based on an instance segmentation model to obtain a segmentation mask. Specifically, the second target region is input into the YOLO-Seg instance segmentation model, which performs pixel-level segmentation on the second target region and outputs the mask contour of each target. A segmentation mask is generated based on the mask contour. Subsequently, the tobacco brand classification device performs deduplication and counting based on the segmentation mask and obtains the target sub-image with the background removed. In the deduplication and counting stage, the minimum bounding rectangle of the corresponding target is calculated based on the segmentation mask. The aspect ratio feature is calculated based on the minimum bounding rectangle. The overlapping masks are deduplicated by combining the mask instance count and the Non-Maximum Suppression (NMS) algorithm. The target count is completed based on the number of deduplicated mask instances. In the clean image acquisition stage, a guided cropping box is generated based on the contour of the segmentation mask. The guided cropping box is used to crop the second target region, removing background pixels. The cropped clean single-product image is used as the target sub-image. Through this series of preprocessing steps, the system not only accurately counts the number of cigarette products in densely stacked scenarios, but also eliminates background interference information such as shelves and adjacent cigarette products, providing high-quality, clean image data for subsequent feature extraction.

[0197] After image preprocessing, the tobacco brand classification device uses a fine-grained feature extraction model to extract features from the regions of the first target region excluding the second target region and the target sub-images, obtaining feature vectors. Specifically, the DINO pre-trained model is used to extract features from the regions of the first target region excluding the second target region and the target sub-images. The extracted raw feature vectors are then L2 normalized, and invalid Regions of Interest (ROIs) features that are too small or completely black are filtered out, retaining only the effective high-dimensional feature vectors. Whether it is a directly cropped regular region or a clean sub-image after segmentation and background removal, it is uniformly transformed into a high-dimensional feature vector representation in this step, ensuring the consistency of the feature space.

[0198] Subsequently, the tobacco brand classification device matches the feature vectors with a standard vector library and outputs the classification results. Specifically, the device calculates the similarity between the feature vectors and each standard vector in the library. If the similarity is greater than or equal to a preset threshold, the corresponding brand or specification classification result is output; if the similarity is less than the preset threshold, an unknown category result is output. For multiple candidate classification results with similarity differences less than a preset threshold, the aspect ratio feature corresponding to the segmentation mask is used for auxiliary disambiguation to determine the final classification result from the multiple candidate results. For example, when the packaging patterns and colors of two cigarette brands are highly similar, resulting in extremely close cosine similarity of the feature vectors, the system retrieves the previously calculated aspect ratio feature as an auxiliary judgment condition, using the fixed geometric differences in the physical dimensions of different cigarette specifications for accurate differentiation. The tobacco brand classification device quickly completes the above matching process through the FAISS vector retrieval engine, ultimately outputting structured classification results for each cigarette product, including brand, specification, quantity, and placement.

[0199] After outputting the classification results, the tobacco brand classification device aggregates the classification results from multiple tobacco images to generate detailed inventory data. The system groups and aggregates the recognition results according to dimensions such as store, manufacturer, cigarette brand, and specification, automatically calculating the shelf availability and product proportion of each type of cigarette. Subsequently, the detailed inventory data is automatically written to the asset ledger system through a standardized data interface. For example, the tobacco brand classification device automatically initiates write requests to the asset ledger system via a REST API interface, eliminating the need for manual data entry. To ensure data security, the system employs an append-only logging mechanism to store the write operation trajectory and modification records of the detailed inventory data. Every inventory count, data modification, and inbound / outbound operation trajectory is completely retained and cannot be tampered with, achieving full traceability of inventory records. The tobacco brand classification device ultimately transmits the identification and labeling information and inventory statistics results back to the mobile device, where operators can review them on the front-end page. The entire inventory process, from image acquisition to ledger updates, achieves end-to-end fully automated closed loop. The time for a single inventory check has been reduced from hours in the traditional manual mode to minutes, significantly reducing labor costs and human error rates.

[0200] Based on the descriptions of the above embodiments, this application achieves high-precision automated identification of tobacco products by introducing a preset difficult-to-identify target library for dual-path diversion, and combining instance segmentation for deduplication and background removal with fine-grained feature unified retrieval.

[0201] Example 11: The above primarily describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, such as a tobacco brand sorting device, includes at least one of the hardware structures and software modules corresponding to the execution of each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0202] This application embodiment can divide the tobacco brand sorting device into functional units according to the above method example. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or software. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0203] When using integrated units, Figure 9 A possible structural schematic diagram of the tobacco brand classification device (referred to as tobacco brand classification device 90) involved in the above embodiments is shown. The tobacco brand classification device 90 includes a processing unit 901 and a communication unit 902, and may also include a storage unit 903. Figure 9 The schematic diagram shown can be used to illustrate the structure of the tobacco brand classification device involved in the above embodiments.

[0204] when Figure 9 The schematic diagram shown illustrates the structure of the tobacco brand sorting device involved in the above embodiments. The processing unit 901 is used to control and manage the operation of the tobacco brand sorting device, the communication unit 902 is used for the tobacco brand sorting device to communicate with other devices, and the storage unit 903 is used to store the program code and data of the tobacco brand sorting device.

[0205] The processing unit 901 can be a processor or a controller, and the communication unit 902 can be a communication interface, transceiver, transceiver circuit, transceiver device, etc. The term "communication interface" is a general term and may include one or more interfaces. The storage unit 903 can be a memory. When the tobacco brand sorting device 90 is a chip, the processing unit 901 can be a processor or a controller, and the communication unit 902 can be an input interface and / or an output interface, pins, or circuits, etc. The storage unit 903 can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., read-only memory (ROM), random access memory (RAM, etc.).

[0206] The communication unit can also be called a transceiver unit. The antenna and control circuit with transceiver functions in the tobacco brand sorting device 90 can be considered as the communication unit 902 of the tobacco brand sorting device 90, and the processor with processing functions can be considered as the processing unit 901 of the tobacco brand sorting device 90. Optionally, the device in the communication unit 902 used to implement the receiving function can be considered as a communication unit. The communication unit is used to execute the receiving steps in the embodiments of this application, and the communication unit can be a receiver, a receiver circuit, etc. The device in the communication unit 902 used to implement the transmitting function can be considered as a transmitting unit. The transmitting unit is used to execute the transmitting steps in the embodiments of this application, and the transmitting unit can be a transmitter, a transmitter, a transmitting circuit, etc.

[0207] Figure 9 If the integrated units in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0208] Figure 9 The units in the process can also be called modules; for example, a processing unit can be called a processing module.

[0209] This application also provides a hardware structure diagram of a tobacco brand sorting device (denoted as tobacco brand sorting device 100), see [link to diagram]. Figure 10 The tobacco brand sorting device 100 includes a processor 1001, and optionally, a memory 1002 connected to the processor 1001.

[0210] In the first possible implementation, see Figure 10 The tobacco brand sorting device 100 also includes a transceiver 1003. The processor 1001, memory 1002, and transceiver 1003 are connected via a bus. The transceiver 1003 is used to communicate with other devices or communication networks. Optionally, the transceiver 1003 may include a transmitter and a receiver. The device in the transceiver 1003 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of this application. The device in the transceiver 1003 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of this application.

[0211] Based on the first possible implementation method Figure 10 The schematic diagram shown can be used to illustrate the structure of the tobacco brand classification device involved in the above embodiments.

[0212] in, Figure 10 This can also be illustrated by the system chip in the tobacco brand sorting device. In this case, the actions performed by the tobacco brand sorting device can be implemented by this system chip; the specific actions performed can be found above and will not be repeated here.

[0213] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0214] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., which are various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a separate semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may be integrated with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits) to form a SoC (System-on-a-Chip), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0215] The memory in the embodiments of this application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto.

[0216] This application also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0217] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0218] This application also provides a chip including a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.

[0219] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0220] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0221] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A method for classifying tobacco brands, characterized in that, include: Acquire multiple tobacco images; The target region is obtained by performing target detection on the multiple tobacco images based on the target detection model. The first target region is matched with a preset database of difficult-to-identify targets to determine the second target region; The second target region is segmented based on the instance segmentation model to obtain a segmentation mask; Based on the segmentation mask, perform deduplication counting and obtain the target sub-image with background removed; Based on the fine-grained feature extraction model, fine-grained feature extraction is performed on the regions in the first target region other than the second target region and the target sub-image to obtain feature vectors; The feature vector is matched and retrieved against a standard vector library to output the classification result.

2. The method according to claim 1, characterized in that, The step of performing target detection on the multiple tobacco images based on the target detection model to obtain a first target region includes: The multiple tobacco images are input into the YOLOv11 target detection model; The multiple tobacco images are processed by high-resolution adaptation and non-maximum suppression (NMS) parameter adjustment to output the bounding boxes and detection confidence scores of all tobacco products. The region of interest (ROI) sub-map for each tobacco product is obtained by cropping based on the bounding box, and serves as the first target region.

3. The method according to claim 1, characterized in that, The step of matching the first target region with a preset difficult-to-identify target database to determine the second target region includes: Obtain the coarse classification result of the first target region; The coarse classification results are matched with the difficult-to-identify category tags in the preset difficult-to-identify target library; If the coarse classification result matches any difficult-to-identify category label in the preset difficult-to-identify target library, then the corresponding first target region is determined as the second target region.

4. The method according to claim 1, characterized in that, The deduplication counting based on the segmentation mask includes: Calculate the minimum bounding rectangle of the corresponding target based on the segmentation mask; Calculate the aspect ratio feature based on the minimum bounding rectangle; The overlapping mask is deduplicated by combining the number of mask instances and nonmaximum suppression (NMS). The target count is completed based on the number of deduplicated mask instances.

5. The method according to claim 4, characterized in that, The process of obtaining the target sub-image with the background removed includes: A guide clipping frame is generated based on the outline of the segmentation mask; The second target area is cropped using the guided cropping frame to remove background pixels; The cropped, clean single-item image is used as the target sub-image.

6. The method according to claim 1, characterized in that, The fine-grained feature extraction model performs fine-grained feature extraction on the regions in the first target region other than the second target region and the target sub-image to obtain feature vectors, including: The DINO model is used to extract features from the regions of the first target region other than the second target region and the target sub-image. The extracted original feature vectors are subjected to L2 normalization. Filter out invalid Regions of Interest (ROIs) features that are too small or completely black, and retain the valid high-dimensional feature vectors as the feature vectors.

7. The method according to claim 6, characterized in that, The step of matching and retrieving the feature vector with a standard vector library and outputting classification results includes: Calculate the similarity between the feature vector and each standard vector in the standard vector library; If the similarity is greater than or equal to a preset threshold, the corresponding brand or specification classification result will be output. If the similarity is less than the preset threshold, then an unknown category result is output; For multiple candidate classification results with similarity differences less than a preset difference threshold, auxiliary disambiguation is performed by combining the aspect ratio features corresponding to the segmentation mask, and the final classification result is determined from the multiple candidate classification results.

8. The method according to claim 1, characterized in that, The step of performing instance segmentation on the second target region based on the instance segmentation model to obtain a segmentation mask includes: The second target region is input into the YOLO-Seg instance segmentation model to perform pixel-level segmentation on the second target region and output the mask contour of each target. The segmentation mask is generated based on the mask contour.

9. The method according to claim 7, characterized in that, After outputting the classification result, the method further includes: The classification results of the multiple tobacco images are summarized to generate detailed inventory data; The inventory count details are automatically written into the asset ledger system through a standardized data interface. An append-only log recording mechanism is used to store the write operation trajectory and modification records of the inventory count details.

10. A tobacco brand sorting device, characterized in that, The device includes: a communication unit and a processing unit; The communication unit is used to acquire multiple tobacco images; The processing unit is configured to: perform target detection on the multiple tobacco images based on a target detection model to obtain a first target region; match the first target region with a preset difficult-to-identify target library to determine a second target region; perform instance segmentation on the second target region based on an instance segmentation model to obtain a segmentation mask; perform deduplication counting based on the segmentation mask and obtain a target sub-image with background removed; perform fine-grained feature extraction on the regions of the first target region other than the second target region and the target sub-image based on a fine-grained feature extraction model to obtain a feature vector; and perform matching and retrieval of the feature vector with a standard vector library to output a classification result.

Images