Cigarette packet inverted image recognition method and device based on comparative learning model and electronic equipment

Abstract

The invention provides a cigarette packet inverted image recognition method and device based on a comparative learning model, and electronic equipment. The method comprises the following steps: obtaining a to-be-recognized cigarette packet image from a whole cigarette packet image based on MSDA-YOLO, a convolutional neural network and a feature pyramid network; determining a distance between the to-be-identified cigarette packet image and the cigarette packet inverted image based on a pre-trained contrast learning model, and determining a cigarette packet type of the to-be-identified cigarette packet image according to the distance and a distance threshold; if the cigarette packet type of the to-be-identified cigarette packet image is a cigarette packet inverted image, neglecting the to-be-identified cigarette packet image; and if the cigarette packet type of the to-be-identified cigarette packet image is a cigarette packet body image, identifying and acquiring cigarette packet information in the to-be-identified cigarette packet image, and performing cigarette packet quantity statistics based on the cigarette packet information. Therefore, the to-be-identified cigarette packet image is input into the pre-trained contrast learning model, so that the inverted image of the cigarette packet can be accurately identified, and the inverted image interference during counting of the number of the cigarette packets is effectively eliminated.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a cigarette package reflection recognition method, device and electronic equipment based on a contrast learning model. Background Art

[0002] In smart retail and vending scenarios, customers interact with products more frequently, and accurate product management and inventory control have become the key to improving customer satisfaction and operational efficiency. However, when products such as cigarette packs are carefully displayed in a retail environment equipped with reflective surfaces (such as glass counters, stainless steel display racks, or any display platform made of smooth materials), technical challenges also arise. System cameras are the eyes of these smart retail systems, responsible for capturing and identifying products on the shelves, but under such specific conditions, they may indiscriminately record all images, including the real image of the cigarette pack and its mirror reflection.

[0003] Although traditional cigarette pack recognition technology can identify and count products on the shelf to a certain extent, it often cannot effectively distinguish which are real images of cigarette packs and which are reflected images caused by light refraction or reflection. This technical bottleneck leads to a common problem: the system will mistakenly identify the reflected images of cigarette packs as independent, real cigarette packs, which will cause deviations in the subsequent inventory statistics and management process.

[0004] This kind of error in quantity statistics is extremely disadvantageous to merchants. Not only will it make it impossible for merchants to accurately grasp the real-time inventory status of cigarette packs, it may also trigger a series of chain reactions, such as over-replenishment or out-of-stock phenomena due to inaccurate inventory information, which in turn affects customers' shopping experience and trust in the brand. In addition, incorrect inventory data may also mislead merchants' sales strategies and market forecasts, which will have a negative impact on the company's operating efficiency and profitability in the long run. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present application provides a cigarette pack reflection recognition method, device and electronic device based on a contrastive learning model, aiming to solve the technical problem that the prior art cannot accurately distinguish between the reflection image and the real image of the cigarette pack, thus causing errors in the number of cigarette packs. Specifically: In the first aspect, an embodiment of the present application provides a method for identifying cigarette package reflections based on a contrastive learning model, the method comprising: obtaining a cigarette package image to be identified from the overall image of the cigarette package based on MSDA-YOLO, a convolutional neural network and a feature pyramid network; determining the distance between the cigarette package image to be identified and the cigarette package reflection image based on a pre-trained contrastive learning model, and determining the cigarette package type of the cigarette package image to be identified according to the distance and the distance threshold; if the cigarette package type of the cigarette package image to be identified is the cigarette package reflection image, ignoring the cigarette package image to be identified; if the cigarette package type of the cigarette package image to be identified is the cigarette package body image, identifying and obtaining the cigarette package information in the cigarette package image to be identified, and performing cigarette package quantity statistics based on the cigarette package information.

[0006] In the second aspect, an embodiment of the present application provides a cigarette package reflection recognition device based on a contrastive learning model, the device comprising: an image acquisition module, used to acquire a cigarette package image to be recognized from the overall image of the cigarette package based on MSDA-YOLO, a convolutional neural network and a feature pyramid network; a recognition module, used to determine the distance between the cigarette package image to be recognized and the cigarette package reflection image based on a pre-trained contrastive learning model, and determine the cigarette package type of the cigarette package image to be recognized according to the distance and the distance threshold; a first judgment module, used to ignore the cigarette package image to be recognized if the cigarette package type of the cigarette package image to be recognized is the cigarette package reflection image; a second judgment module, used to identify and acquire the cigarette package information in the cigarette package image to be recognized if the cigarette package type of the cigarette package image to be recognized is the cigarette package body image, and perform cigarette package quantity statistics based on the cigarette package information.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: one or more processors, a memory, and one or more applications. The one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method of the first aspect.

[0008] In the technical solution provided in the present application, the image of the cigarette package to be identified is obtained from the whole image of the cigarette package based on MSDA-YOLO, convolutional neural network and feature pyramid network; the distance between the image of the cigarette package to be identified and the image of the cigarette package reflection is determined based on the pre-trained contrast learning model, and the type of the cigarette package of the image to be identified is determined according to the distance and the distance threshold; if the type of the cigarette package of the image to be identified is the image of the cigarette package reflection, the image to be identified is ignored; if the type of the cigarette package of the image to be identified is the image of the cigarette package body, the cigarette package information in the image to be identified is identified, and the number of cigarette packages is counted based on the cigarette package information. Therefore, by inputting the image of the cigarette package to be identified into the pre-trained contrast learning model, the reflection image of the cigarette package can be accurately identified; at the same time, when the reflection image of the cigarette package is identified, it is automatically ignored, which effectively eliminates the interference of reflection when counting the number of cigarette packages. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments and drawings obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0010] Figure 1 A flow chart of a cigarette pack reflection recognition method based on a contrastive learning model proposed in an embodiment of the present application is shown.

[0011] Figure 2 A structural block diagram of a cigarette pack reflection recognition device based on a contrastive learning model proposed in an embodiment of the present application is shown.

[0012] Figure 3 A structural block diagram of an electronic device proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0015] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0016] It should be further understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0017] In smart retail and vending scenarios, customers interact with products more frequently, and accurate product management and inventory control have become the key to improving customer satisfaction and operational efficiency. However, when products such as cigarette packs are carefully displayed in a retail environment equipped with reflective surfaces (such as glass counters, stainless steel display racks, or any display platform made of smooth materials), technical challenges also arise. System cameras are the eyes of these smart retail systems, responsible for capturing and identifying products on the shelves, but under such specific conditions, they may indiscriminately record all images, including the real image of the cigarette pack and its mirror reflection.

[0018] Although traditional cigarette pack recognition technology can identify and count products on the shelf to a certain extent, it often cannot effectively distinguish which are real images of cigarette packs and which are reflected images caused by light refraction or reflection. This technical bottleneck leads to a common problem: the system will mistakenly identify the reflected images of cigarette packs as independent, real cigarette packs, which will cause deviations in the subsequent inventory statistics and management process.

[0019] This kind of error in quantity statistics is extremely disadvantageous to merchants. Not only will it make it impossible for merchants to accurately grasp the real-time inventory status of cigarette packs, it may also trigger a series of chain reactions, such as over-replenishment or out-of-stock phenomena due to inaccurate inventory information, which in turn affects customers' shopping experience and trust in the brand. In addition, incorrect inventory data may also mislead merchants' sales strategies and market forecasts, which will have a negative impact on the company's operating efficiency and profitability in the long run.

[0020] Therefore, in order to solve the problems existing in the prior art, the present application proposes a cigarette package reflection recognition method, device and electronic device based on a contrast learning model, which will be described below in conjunction with specific embodiments.

[0021] See also Figure 1 , Figure 1A cigarette pack reflection recognition method based on a contrastive learning model provided in an embodiment of the present application is shown, specifically: Step S110, obtaining a cigarette package image to be identified from the overall image of the cigarette package based on MSDA-YOLO, convolutional neural network and feature pyramid network.

[0022] In an embodiment of the present application, the overall image of a cigarette package refers to a complete image containing one or more cigarette packages and possible other background elements (such as shelves, reflective surfaces, other commodities, etc.). The overall image of the cigarette package can be a real-time image captured by a camera or other image acquisition device, or it can be an image obtained from other channels (such as an image manually uploaded by a merchant); the image of a cigarette package to be identified refers to an image of a single cigarette package separated from the overall image of the cigarette package and prepared for further identification processing.

[0023] Specifically, after the camera obtains the overall image of the cigarette pack, it can cut and separate the cigarette pack in the overall image according to MSDA-YOLO, convolutional neural network and feature pyramid network to form a single cigarette pack image to be identified.

[0024] In some embodiments, a cigarette package image to be identified is obtained from a whole image of a cigarette package based on MSDA-YOLO, a convolutional neural network, and a feature pyramid network, including: obtaining a whole image of a cigarette package, and performing dedistortion processing on the whole image of the cigarette package to obtain a dedistorted whole image of the cigarette package; preprocessing the dedistorted whole image of the cigarette package, detecting the cigarette package image in the preprocessed dedistorted whole image of the cigarette package based on MSDA-YOLO, and determining a cigarette box region image based on the detection result; extracting image features in the cigarette box region image based on a convolutional neural network; inputting the image features into a feature pyramid network, determining a cutting region from the dedistorted whole image of the cigarette package based on the multi-scale feature information and attention mechanism output by the feature pyramid network, and cutting the cutting region based on DeepLab to obtain a cigarette package cutting image; and straightening the obtained cigarette package cutting image to obtain a cigarette package image to be identified.

[0025] Specifically, in the process of processing the cigarette pack image, we first obtain the whole image of the cigarette pack containing at least the cigarette pack. Due to factors such as the image acquisition device or the environment, the image may be distorted, so it is necessary to perform dedistortion processing on it to restore the original shape and proportion of the image and obtain the dedistorted whole image of the cigarette pack. Next, in order to improve the effect of subsequent processing, the dedistorted whole image is first preprocessed, and then the preprocessed dedistorted whole image of the cigarette pack is scanned by MSDA-YOLO to detect and identify the approximate position and range of the cigarette pack image, so that we can cut out irrelevant scenes from the dedistorted whole image of the cigarette pack to obtain the cigarette box area image. Among them, the preprocessing can be, for example, image normalization, image enhancement (such as contrast adjustment, noise suppression) and other operations. Further, in order to more deeply analyze the specific position of each cigarette pack in the cigarette box area image, a convolutional neural network is used to extract the key features in the image. Inputting these image features into the feature pyramid network can more accurately understand the cigarette pack information in the image, and then accurately determine the cutting area from the dedistorted whole image of the cigarette pack according to the output multi-scale feature information and the attention map generated by the attention mechanism. Then, DeepLab is used to perform fine pixel-level segmentation processing on the cut area. A single cigarette pack cut image is accurately separated from the overall dedistorted cigarette pack image, and finally the cigarette pack cut image is corrected by a preset image straightening processing algorithm to keep it vertical, and finally the cigarette pack image to be identified is obtained. Among them, the preset image straightening processing algorithm can be, for example, affine transformation, perspective transformation, etc.

[0026] Furthermore, after obtaining the cut cigarette pack image from the de-distorted whole cigarette pack image based on the multi-scale feature information output by the feature pyramid network and the attention mechanism, since cigarette packs may be occluded in actual applications, in order to ensure that the number of cigarette packs is as accurate as possible, it is also necessary to identify and count the occluded cigarette packs. Specifically, the occluded cigarette pack image in the pre-processed de-distorted whole cigarette pack image is obtained based on U-Net; the occluded cigarette pack image is reconstructed based on the generative adversarial network and the context-aware mechanism to obtain the reconstructed cigarette pack image; the cut cigarette pack image and the reconstructed cigarette pack image are straightened to obtain the cigarette pack image to be identified.

[0027] Specifically, if there is a cigarette pack occlusion, U-Net can be used to segment the cigarette pack occluded area from the preprocessed de-distorted overall image of the cigarette pack, and then the occluded area is reconstructed based on the generative adversarial network (GAN) and context-aware mechanism to generate a reconstructed cigarette pack image; then the cigarette pack cut image and the cigarette pack reconstructed image are straightened to obtain all the cigarette pack images to be identified from the overall image of the cigarette pack.

[0028] In some embodiments, since the shapes of the cigarette packs on the left and right sides are roughly the same during the selling process, after the cigarette pack reconstructed image is obtained, the credibility of the cigarette pack reconstructed image can be determined based on the similarity between the cigarette pack reconstructed image and the cigarette pack image on at least one side thereof. If the similarity is lower than a preset threshold, it is considered that the reconstructed image has a large deviation from the actual image and the reconstruction fails. Subsequently, a separate secondary reconstruction can be performed according to its position as needed, or it can be directly discarded.

[0029] Step S120, determining the distance between the cigarette package image to be identified and the cigarette package reflection image based on the pre-trained contrast learning model, and determining the cigarette package type of the cigarette package image to be identified according to the distance and the distance threshold.

[0030] In the embodiment of the present application, the cigarette package type includes a cigarette package body image and a cigarette package reflection image.

[0031] Specifically, if the distance is greater than or equal to the distance threshold, the cigarette package type of the cigarette package image to be identified is considered to be a cigarette package reflection image; if the distance is less than the distance threshold, the cigarette package type of the cigarette package image to be identified is considered to be a cigarette package body image.

[0032] In an embodiment of the present application, the training process of the contrastive learning model can be: obtaining a cigarette package main body image, a cigarette package reflection image and other cigarette package images to construct a training sample; wherein the other cigarette package images are in the same direction as the cigarette package main body image but in different scenes, and are not cigarette package reflection images; randomly obtaining a cigarette package main body image, other cigarette package images and cigarette package reflection images from the training samples to form a triplet; extracting cigarette package image features in the triplet based on the convolutional neural network in the contrastive learning model, and converting the cigarette package image features into a cigarette package feature vector; determining a first distance between the cigarette package main body image and other cigarette package images based on the cigarette package feature vector and the Euclidean distance, and determining a second distance between the cigarette package main body image and the cigarette package reflection image; determining a loss value corresponding to the triplet according to the first feature vector, the second feature vector and the Triplet Loss; adding the loss values ​​of all triplets to determine a total loss value, updating the weights and biases of the contrastive learning model according to the total loss value based on the back propagation algorithm and the small batch gradient descent algorithm, and iterating until the change in the loss value of the contrastive learning model meets the preset conditions to obtain a trained contrastive learning model.

[0033] In some embodiments, cigarette package images, cigarette package reflection images, and other cigarette package images can be collected from multiple channels to ensure the diversity and representativeness of the training samples, and to cover cigarette package images of different brands, models, and scenes. It is understandable that the initially collected images can be cleaned to remove overly blurred and repeated images, and the cleaned images can be cropped, scaled, and normalized to ensure the consistency of the input data.

[0034] In some embodiments, the specific architecture of the convolutional neural network may be, for example, ResNet, VGG, etc. Specifically, the input layer of the convolutional neural network receives a fixed-size cigarette package body image, a cigarette package reflection image, and other cigarette package images, and then obtains cigarette package image features through a fully connected layer, and finally outputs a fixed-length feature vector.

[0035] In the embodiment of the present application, Triplet Loss is the core loss function of the contrastive learning model, which aims to minimize the distance between positive sample pairs (i.e., the cigarette package image and other images of the cigarette package) and maximize the distance between negative sample pairs (i.e., the cigarette package image and the cigarette package reflection image), while maintaining an interval α to ensure that the distance between negative sample pairs is always greater than the distance between positive sample pairs plus the interval. Specifically, Triplet Loss is defined as follows: ; Among them, f(A) is the embedded representation of the cigarette package image sample, f(P) is the embedded representation of other cigarette package image samples, f(N) is the embedded representation of the cigarette package reflection image sample, and α is a hyperparameter used to control the minimum distance between the cigarette package image sample and the cigarette package reflection image sample.

[0036] In some embodiments, the preset conditions may be, for example, that the loss value no longer decreases significantly, a certain number of iterations is reached, the performance on the validation set is optimal, etc.

[0037] In some embodiments, updating the weights and biases of the contrastive learning model according to the total loss value based on the back propagation algorithm and the mini-batch gradient descent algorithm may include: determining the gradients of the weights and biases of the contrastive learning model based on the total loss value and the back propagation algorithm; adjusting the first learning rate based on the gradients of the weights and biases and the Adam optimizer; updating the weights and biases based on the gradients of the weights and biases, the first learning rate, and the mini-batch gradient descent algorithm.

[0038] Among them. The first learning rate is the initial learning rate. Specifically, during the training phase, the model receives a series of triplet data, and the loss value of each triplet is calculated through forward propagation. These loss values ​​are accumulated to form a total loss value. The total loss value reflects the difference between the model's current prediction ability for all training samples and the actual labels. In order to improve the model, the gradient of the total loss value with respect to the weights and biases of the model can be calculated through the back propagation algorithm. These gradients indicate how the weights and biases should be adjusted to reduce the total loss value. Specifically, the back propagation algorithm starts from the output layer and calculates the gradient of each layer parameter layer by layer until it reaches the input layer. However, updating the weights and biases directly based on these gradients may cause the training process to be unstable or converge too slowly. Therefore, the Adam optimizer is used in this scheme to help adjust the learning rate to control the update step size of the weights and biases. It can automatically adjust the learning rate of each parameter and calculate the update value based on the first-order moment estimate and second-order moment estimate of the gradient. In the Adam optimizer, an initial learning rate (i.e., the first learning rate) is set. The Adam optimizer calculates the exponential moving average of the gradient (i.e., the first-order moment estimate) and the exponential moving average of the square of the gradient (i.e., the second-order moment estimate), and then adjusts the learning rate of each parameter based on these two estimates. Finally, based on the gradients of the weights and biases, the learning rate adjusted by the Adam optimizer, and the mini-batch gradient descent algorithm, the weights and biases of the model are updated. It can be understood that based on the back-propagation algorithm, the mini-batch gradient descent algorithm, and the Adam optimizer, the weights and biases of the comparative learning model can be effectively updated, thereby minimizing the total loss value and improving the performance of the model.

[0039] Furthermore, if the convergence effect is not ideal, it may be that there are not enough negative samples, so the number of negative samples needs to be increased. Specifically, the number of cigarette package reflection images in the training sample is determined to be sufficient according to the change in the loss value of the comparison learning model during the iteration process; if not, data enhancement is performed based on the cigarette package reflection image to obtain a pseudo cigarette package reflection image, and the pseudo cigarette package reflection image is added to the training sample.

[0040] Specifically, if the loss value decreases slowly or stagnates, it means that the number of cigarette package reflection images is insufficient. Therefore, the cigarette package reflection data can be enhanced to obtain pseudo cigarette package reflection images, and then the dimensional cigarette package reflection images are added to the training samples as negative samples. Among them, data enhancement can be, for example, rotating, scaling, flipping, cropping, adding noise and other operations to the cigarette package reflection images to generate pseudo cigarette package reflection images. Although these pseudo images are not real reflections, they can simulate different reflection situations, thereby helping the model learn more useful features. After adding the generated pseudo cigarette package reflection images to the training samples, the comparative learning model can continue to be trained. As the training progresses, the model will be able to access more diverse negative samples, thereby enhancing its ability to distinguish between the cigarette package itself and the reflection, which will help improve the convergence effect of the model and ultimately improve the recognition accuracy and generalization ability of the model.

[0041] Furthermore, on the basis of at least one training, key training samples can be obtained based on historical training results to focus on training samples that are difficult to distinguish. Specifically, based on the number of triplets and the overall loss value, a preset number of triplets with the largest loss value are obtained; data enhancement is performed on the obtained triplets to obtain key training samples; the second learning rate is adjusted based on the number of key training triplets in the key training samples, the total loss value, and the change in the total loss value; the weights and biases of the comparison learning model are updated based on the total loss value, the back propagation algorithm, the second learning rate, and the small batch gradient descent algorithm.

[0042] Specifically, the loss values ​​of all triplets during the training process are first calculated, and then according to the size of the loss value, a preset number of triplets with the largest loss values ​​(for example, the top 10% or the top N) are selected as the key training objects. These triplets usually represent samples that are currently difficult for the model to distinguish, so focusing on them is expected to significantly improve the performance of the model. Then, data augmentation techniques are used to generate more variants of the images in the triplets. Data augmentation can include various image transformation operations, such as rotation, scaling, cropping, flipping, adding noise, and possible color transformation, brightness adjustment, etc. Through data augmentation, multiple similar samples can be generated for each key triplet, thereby forming a rich set of key training samples. Then, the second learning rate is dynamically adjusted according to the number of key training triplets (i.e., the richness of key training samples), the total loss value (i.e., the overall performance of the current model), and the change in the total loss value (i.e., the speed of improvement of the model performance). For example, if the total loss value is still high and decreases slowly, we can increase the second learning rate to accelerate the learning of the key training samples; on the contrary, if the total loss value is already low and tends to be stable, we can appropriately reduce the second learning rate to prevent overfitting. After obtaining the key training samples and adjusting the second learning rate, the loss value of the key training samples is calculated first, and the weighted sum is performed with the total loss value. Then, the back propagation algorithm is used to calculate the gradient of the total loss value with respect to the model weights and biases. Then, the step size of these gradients is adjusted according to the second learning rate, and finally, the mini-batch gradient descent algorithm is used to update the model weights and biases. In this way, while maintaining the stability of the model, the training can be focused on those samples that are difficult to distinguish, thereby improving the overall performance of the model.

[0043] The initial second learning rate is greater than the initial first learning rate. In some embodiments, the initial second learning rate may also be determined based on an average value of loss values ​​in a preset number of triples obtained.

[0044] In some embodiments, the cigarette package feature vector can also be determined based on the fusion feature. Specifically, the cigarette package image features in the triplet are extracted based on the convolutional neural network in the contrastive learning model; the physical image features of the cigarette package are determined based on the light source category and light source position corresponding to the cigarette package body image, other cigarette package images, and cigarette package reflection image in the triplet; based on the convolutional neural network, the cigarette package feature vector is determined based on the cigarette package image features, the first weight, the cigarette package physical image features, and the second weight.

[0045] The light source type may be, for example, daylight, fluorescent light, LED, etc., and the light source position may be, for example, a front light source, a side light source, a backlight, etc.

[0046] Specifically, by combining the previously extracted image features with the physical image features, a comprehensive cigarette package feature vector can be formed. This vector contains not only the cigarette package image features of the image itself, but also the physical image features of the cigarette package (lighting, angle, position, etc.). According to the actual sales situation, since the physical features of the cigarette package have more uncertainties, a first weight can be assigned to the cigarette package image features, and a second weight can be assigned to the physical image features of the cigarette package. The second weight is at most half of the first weight. It can be understood that since the cigarette package image features and the physical image features of the cigarette package are fused into a comprehensive feature, the fused features need to be expanded into a one-dimensional vector before they can be input into the convolutional neural network to determine the cigarette package feature vector.

[0047] It is understandable that combining the image features extracted by the convolutional neural network with the physical image features of the cigarette package and assigning appropriate weights to them to form the final cigarette package feature vector has brought significant benefits in many aspects: 1. Improving the richness and accuracy of feature expression: Although the image features extracted by the convolutional neural network alone can capture the high-level information of the image, they may ignore the physical conditions directly related to the image generation (such as lighting, angle, etc.). By introducing the physical image features of the cigarette package, such as the type and position of the light source, this deficiency can be compensated, so that the feature vector can more comprehensively reflect the essential characteristics of the cigarette package. This combination makes the feature vector contain both the detailed information of the image itself and the contextual information of the image generation, thereby improving the richness and accuracy of the feature expression. 2. Enhance the robustness and generalization ability of the model: In practical applications, cigarette package images may be affected by various factors, such as lighting changes, angle changes, occlusion, etc. By introducing physical image features, the model can better understand the impact of these factors on the image, so as to more accurately identify cigarette packages. This ability makes the model more robust when facing complex and changing actual scenes. At the same time, because the feature vector contains more comprehensive information, the model can learn more useful feature combinations and patterns during training, thereby improving its generalization ability on new samples.

[0048] It should be noted that the loss value acquisition method, feature extraction method, etc. mentioned in the above content can all be explained with reference to the ideas of the first explanation, and the input data that needs to be adjusted can be adjusted accordingly.

[0049] Step S130: If the cigarette package type of the cigarette package image to be identified is a cigarette package reflection image, the cigarette package image to be identified is ignored.

[0050] In the embodiment of the present application, when the contrast learning model identifies that the cigarette package type of the cigarette package image to be identified is a cigarette package reflection image, the cigarette package image to be identified is ignored and output for identification and quantity statistics are not performed. This mechanism ensures that the cigarette package reflection image does not interfere with the final recognition result, avoids the problem of misidentification and repeated statistics, and improves the accuracy of recognition and the stability of the system.

[0051] Furthermore, ignoring reflected images means that the model does not need to perform complex output recognition on these images. This can reduce the consumption of computing resources, improve processing speed, and enable the model to process other more valuable images faster; at the same time, ignoring reflected images can simplify the processing flow. There is no need for additional labeling, classification, or filtering operations on these images, which reduces the workload of data processing and improves work efficiency; in addition, in practical applications, if the model mistakenly identifies the reflected image as the cigarette package itself or other types, it may cause confusion and false alarms. This may lead to subsequent wrong decisions or unnecessary interventions. By ignoring reflected images, such confusion and false alarms can be avoided, improving the stability and reliability of the system.

[0052] Step S140: If the cigarette package type of the cigarette package image to be identified is a cigarette package body image, the cigarette package information in the cigarette package image to be identified is identified and obtained, and the number of cigarette packages is counted based on the cigarette package information.

[0053] In the embodiment of the present application, if it is determined that the cigarette package type of the cigarette package image to be identified is a cigarette package body image, the text and pictures in the cigarette package body image are identified to determine the cigarette brand, specifications, etc. to which the cigarette package image to be identified belongs, and then the number of cigarette packages is counted based on the identified cigarette package information. The cigarette package body image is a real image of the cigarette package.

[0054] In some embodiments, after the recognition and statistics are completed, feedback data of the image to be recognized can be determined based on MSDA-YOLO; wherein the feedback data includes cutting accuracy and reconstruction effect; MSDA-YOLO, DeepLab and generative adversarial network are adjusted according to the feedback data, and the adjusted MSDA-YOLO, DeepLab and generative adversarial network are used as new MSDA-YOLO, DeepLab and generative adversarial network.

[0055] Specifically, after each recognition and statistical task is completed, MSDA-YOLO can be used to evaluate the effect of this task and generate feedback data. According to the collected feedback data, the MSDA-YOLO, DeepLab and generative adversarial network required in this application can be adjusted and optimized, and then the adjusted MSDA-YOLO, DeepLab and generative adversarial network will be used the next time the cigarette package reflection recognition method provided in this application is executed. Among them, the adjustment may include fine-tuning of model parameters, improvement of network structure, enhancement of training data, etc. Each time the cigarette package reflection recognition method is optimized, the accuracy and efficiency of subsequent recognition can be gradually improved.

[0056] In the technical solution provided in the present application, the image of the cigarette package to be identified is obtained from the whole image of the cigarette package based on MSDA-YOLO, convolutional neural network and feature pyramid network; the distance between the image of the cigarette package to be identified and the image of the cigarette package reflection is determined based on the pre-trained contrast learning model, and the type of the cigarette package of the image to be identified is determined according to the distance and the distance threshold; if the type of the cigarette package of the image to be identified is the image of the cigarette package reflection, the image to be identified is ignored; if the type of the cigarette package of the image to be identified is the image of the cigarette package body, the cigarette package information in the image to be identified is identified, and the number of cigarette packages is counted based on the cigarette package information. Therefore, by inputting the image of the cigarette package to be identified into the pre-trained contrast learning model, the reflection image of the cigarette package can be accurately identified; at the same time, when the reflection image of the cigarette package is identified, it is automatically ignored, which effectively eliminates the interference of reflection when counting the number of cigarette packages.

[0057] See also Figure 2 , Figure 2 A cigarette package reflection recognition device 100 based on a contrast learning model provided in an embodiment of the present application is shown, and the device includes an image acquisition module 110, a recognition module 120, a first judgment module 130, and a second judgment module 140. Specifically: An image acquisition module 110 is used to acquire a cigarette package image to be identified from the whole cigarette package image based on MSDA-YOLO, convolutional neural network and feature pyramid network; The recognition module 120 is used to determine the distance between the cigarette package image to be recognized and the cigarette package reflection image based on the pre-trained contrast learning model, and determine the cigarette package type of the cigarette package image to be recognized according to the distance and the distance threshold; A first judgment module 130 is used to ignore the cigarette package image to be identified if the cigarette package type of the cigarette package image to be identified is a cigarette package reflection image; The second judgment module 140 is used to identify and obtain cigarette package information in the cigarette package image to be identified if the cigarette package type of the cigarette package image to be identified is a cigarette package body image, and to count the number of cigarette packages based on the cigarette package information.

[0058] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Any processing method described in the method embodiment can be implemented by the corresponding processing module in the device embodiment, and will not be repeated one by one in the device embodiment.

[0059] See also Figure 3 , Figure 3 An electronic device 200 proposed in an embodiment of the present application is shown. The electronic device 200 in the present application may include one or more of the following components: a processor 210, a memory 220, and one or more applications, wherein the one or more applications may be stored in the memory 220 and configured to be executed by the one or more processors 210, and the one or more applications are configured to execute the method described in the foregoing method embodiment.

[0060] The processor 210 may include one or more processing cores. The processor 210 uses various interfaces and lines to connect various parts of the entire electronic device 200, and executes various functions and processes data of the electronic device 200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 220, and calling data stored in the memory 220. Optionally, the processor 210 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 210 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 210, but may be implemented separately through a communication chip.

[0061] The memory 220 may include a random access memory (RAM) or a read-only memory (ROM). The memory 220 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as an identification function, a judgment function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the electronic device 200 during use, such as a cigarette package image to be identified, a cigarette package reflection image, etc.

[0062] The present application provides a method, device and electronic device for identifying cigarette package reflections based on a contrastive learning model. The method obtains a cigarette package image to be identified from the overall image of the cigarette package based on MSDA-YOLO, a convolutional neural network and a feature pyramid network; determines the distance between the cigarette package image to be identified and the cigarette package reflection image based on a pre-trained contrastive learning model, and determines the cigarette package type of the cigarette package image to be identified based on the distance and the distance threshold; if the cigarette package type of the cigarette package image to be identified is a cigarette package reflection image, the cigarette package image to be identified is ignored; if the cigarette package type of the cigarette package image to be identified is a cigarette package body image, the cigarette package information in the cigarette package image to be identified is identified, and the number of cigarette packages is counted based on the cigarette package information. Therefore, by inputting the cigarette package image to be identified into a pre-trained contrastive learning model, the cigarette package reflection image can be accurately identified; at the same time, when the cigarette package reflection image is identified, it is automatically ignored, effectively eliminating the interference of reflections when counting the number of cigarette packages.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A cigarette pack reflection recognition method based on contrastive learning model, characterized in that: The method comprises: Based on MSDA-YOLO, convolutional neural network and feature pyramid network, the cigarette pack image to be identified is obtained from the overall image of the cigarette pack; Determine the distance between the cigarette package image to be identified and the cigarette package reflection image based on a pre-trained contrast learning model, and determine the cigarette package type of the cigarette package image to be identified according to the distance and a distance threshold; If the cigarette package type of the cigarette package image to be identified is the cigarette package reflection image, ignoring the cigarette package image to be identified; If the cigarette package type of the cigarette package image to be identified is the cigarette package body image, the cigarette package information in the cigarette package image to be identified is identified and acquired, and the number of cigarette packages is counted based on the cigarette package information.

2. The method according to claim 1, characterized in that Before obtaining the cigarette package image to be identified from the whole cigarette package image based on MSDA-YOLO, convolutional neural network and feature pyramid network, the method further includes: Obtaining a cigarette package main body image, a cigarette package reflection image, and other cigarette package images to construct training samples; wherein the other cigarette package images have the same direction as the cigarette package main body image but different scenes, and are not the cigarette package reflection images; Randomly acquiring the cigarette package main body image, the cigarette package other images and the cigarette package reflection image from the training samples to form a triplet; Extracting cigarette package image features in the triplet based on a convolutional neural network in a contrastive learning model, and converting the cigarette package image features into a cigarette package feature vector; Determine a first distance between the cigarette package main body image and the other images of the cigarette package based on the cigarette package feature vector and the Euclidean distance, and determine a second distance between the cigarette package main body image and the cigarette package reflection image; Determine a loss value corresponding to the triplet according to the first eigenvector, the second eigenvector and Triplet Loss; The loss values ​​of all the triplets are added together to determine a total loss value, and the weights and biases of the contrastive learning model are updated according to the total loss value based on the back propagation algorithm and the mini-batch gradient descent algorithm, and the training contrastive learning model is obtained by iterating until the change in the loss value of the contrastive learning model meets the preset conditions.

3. The method according to claim 2, characterized in that The method of updating the weights and biases of the contrastive learning model according to the total loss value based on the back propagation algorithm and the mini-batch gradient descent algorithm includes: Determining the gradients of the weights and biases of the contrastive learning model based on the total loss value and a back-propagation algorithm; Adjusting a first learning rate based on the gradients of the weights and biases and an Adam optimizer; updating the weights and the biases based on the gradients of the weights and the biases, the first learning rate, and a mini-batch gradient descent algorithm; After updating the weight and the bias based on the gradient of the weight and the bias, the first learning rate and the mini-batch gradient descent algorithm, the method further includes: Determining whether the number of the cigarette pack reflection images in the training sample is sufficient according to the change of the loss value of the contrastive learning model during the iteration process; If it is not enough, data enhancement is performed based on the cigarette package reflection image to obtain a pseudo cigarette package reflection image, and the pseudo cigarette package reflection image is added to the training sample.

4. The method according to claim 3, characterized in that After updating the weights and biases of the contrastive learning model according to the total loss value based on the back propagation algorithm and the mini-batch gradient descent algorithm, the method further includes: Based on the number of triples and the overall loss value, a preset number of triplets with the largest loss value are obtained; Performing data enhancement on the acquired triples to obtain key training samples; Adjusting the second learning rate based on the number of key training triplets in the key training samples, the total loss value, and the change in the total loss value; The weights and biases of the contrastive learning model are updated based on the total loss value, the back propagation algorithm, the second learning rate, and the mini-batch gradient descent algorithm.

5. The method according to claim 4, characterized in that The convolutional neural network in the contrastive learning model extracts the cigarette package image features in the triplet and converts the cigarette package image features into a cigarette package feature vector, including: Extracting cigarette pack image features in the triplet based on a convolutional neural network in a contrastive learning model; Determine the physical image features of the cigarette package based on the light source categories and light source positions corresponding to the cigarette package main body image, the other cigarette package images, and the cigarette package reflection image in the triplet; Based on the convolutional neural network, a cigarette pack feature vector is determined according to the cigarette pack image feature, the first weight, the cigarette pack physical image feature and the second weight.

6. The method according to any one of claims 2 to 5, characterized in that: The method of obtaining a cigarette package image to be identified from the whole cigarette package image based on MSDA-YOLO, convolutional neural network and feature pyramid network includes: Acquire an overall image of a cigarette package, and perform a dedistortion process on the overall image of the cigarette package to obtain a dedistorted overall image of the cigarette package; Preprocessing the de-distorted overall image of the cigarette pack, detecting the cigarette pack image in the pre-processed de-distorted overall image of the cigarette pack based on MSDA-YOLO, and determining to obtain a cigarette box region image according to the detection result; Extracting image features in the cigarette box area image based on a convolutional neural network; Inputting the image features into a feature pyramid network, determining a cutting area from the dedistorted whole image of the cigarette pack according to the multi-scale feature information output by the feature pyramid network and an attention mechanism, and cutting the cutting area based on DeepLab to obtain a cigarette pack cutting image; The obtained cigarette packet cutting image is straightened to obtain a cigarette packet image to be identified.

7. The method according to claim 6, characterized in that After obtaining the cigarette pack cut image from the dedistorted whole cigarette pack image based on the multi-scale feature information output by the feature pyramid network and the attention mechanism, the method further includes: Based on U-Net, obtain the cigarette pack occlusion image in the preprocessed cigarette pack dedistorted overall image; Reconstructing the cigarette pack occluded image based on a generative adversarial network and a context-aware mechanism to obtain a cigarette pack reconstructed image; The step of performing straightening processing on the obtained cigarette packet cutting image to obtain the cigarette packet image to be identified includes: The cigarette packet cut image and the cigarette packet reconstructed image are straightened to obtain a cigarette packet image to be identified.

8. The method according to claim 7, characterized in that After counting the number of cigarette packs based on the cigarette pack information, the method further includes: Determining feedback data of the image to be identified based on the MSDA-YOLO; wherein the feedback data includes cutting accuracy and reconstruction effect; The MSDA-YOLO, the DeepLab, and the generative adversarial network are adjusted according to the feedback data, and the adjusted MSDA-YOLO, the DeepLab, and the generative adversarial network are used as new MSDA-YOLO, DeepLab, and the generative adversarial network.

9. A cigarette pack reflection recognition device based on a contrast learning model, characterized in that: The device comprises: An image acquisition module is used to acquire the cigarette package image to be identified from the whole cigarette package image based on MSDA-YOLO, convolutional neural network and feature pyramid network; A recognition module, used to determine the distance between the cigarette package image to be recognized and the cigarette package reflection image based on a pre-trained contrast learning model, and determine the cigarette package type of the cigarette package image to be recognized according to the distance and a distance threshold; a first judgment module, configured to ignore the cigarette package image to be identified if the cigarette package type of the cigarette package image to be identified is the cigarette package reflection image; The second judgment module is used to identify and obtain the cigarette package information in the cigarette package image to be identified if the cigarette package type of the cigarette package image to be identified is the cigarette package body image, and to count the number of cigarette packages based on the cigarette package information.

10. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1-8.

Images