A method, system, terminal, and storage medium for segmenting images of flexible packaged goods.

Abstract

This application provides a method, system, terminal, and storage medium for segmenting images of flexible packaged goods, relating to the field of image segmentation technology. The method includes: acquiring a real-time image of the flexible packaged goods at the moment of being picked up, and preprocessing it to obtain a preprocessed image; based on the preprocessed image, performing feature extraction to obtain first texture features, contour features, and watershed boundary features, and performing feature fusion to obtain fused features; determining whether the preprocessed image is an image of the same type of goods or an image of different types of goods; if it is an image of the same type of goods, inputting the fused features into a semantic segmentation model to obtain a first segmentation result; if it is an image of different types of goods, inputting the fused features into an instance segmentation model to obtain a second segmentation result; obtaining the weight change of the shelf, and verifying the first or second segmentation result; if the verification passes, outputting the corresponding segmentation result. This application can improve the segmentation accuracy of images of multiple flexible packaged goods being bonded together.

Description

Technical Field

[0001] This application relates to the field of image segmentation technology, and in particular to a method, system, terminal, and storage medium for segmenting images of flexible packaging products. Background Technology

[0002] Existing smart self-service vending machines allow products to be freely placed inside. A built-in high-definition camera accurately scans all items, and product recognition is achieved through image processing and visual algorithms. However, when multiple soft-packaged items are taken together, they are prone to sticking together due to static electricity or compression. Traditional image segmentation algorithms, relying on only a single feature, cannot accurately segment the adhesive boundaries, easily misclassifying them as a single item, leading to a high checkout error rate.

[0003] Therefore, how to improve the segmentation accuracy of adhesive images of multi-soft packaged goods is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a method for segmenting images of flexible packaged goods, which can improve the segmentation accuracy of images of multiple flexible packaged goods bonded together. This application also provides a segmentation system, terminal, and storage medium for images of flexible packaged goods, achieving the same technical effects.

[0005] The first objective of this application is to provide a method for segmenting images of flexible packaged goods.

[0006] The aforementioned objective of this application is achieved through the following technical solution:

[0007] A method for segmenting images of soft-packaged goods, applied to self-service vending machines, includes:

[0008] Real-time images of soft-packaged goods are captured at the moment they are picked up, and the real-time images are preprocessed to obtain preprocessed images;

[0009] Based on the preprocessed image, feature extraction is performed to obtain the first texture feature, contour feature, and watershed boundary feature;

[0010] Based on the first texture feature, the contour feature, and the watershed boundary feature, feature fusion is performed to obtain the fused feature;

[0011] Determine whether the preprocessed image is an image of the same type of product or an image of a different type of product;

[0012] If the preprocessed image is a product image of the same type, then the fused features are input into the pre-trained semantic segmentation model to obtain the first segmentation result;

[0013] If the preprocessed image is an image of a different type of product, then the fused features are input into a pre-trained instance segmentation model to obtain a second segmentation result;

[0014] Obtain the weight change of the shelves in the self-service vending machine before and after the soft-packaged goods are taken out. Based on the weight change, verify the first segmentation result or the second segmentation result. If the verification is successful, output the first segmentation result or the second segmentation result.

[0015] Preferably, in the method for segmenting the image of a flexible packaged product, verifying the first segmentation result based on the weight change includes:

[0016] Based on the first segmentation result, determine the quantity of goods;

[0017] Based on the weight change and the quantity of goods, the weight of a single item is calculated, and it is determined whether the weight of a single item matches a preset product weight database.

[0018] If so, then the verification is successful.

[0019] Preferably, in the segmentation method for the image of flexible packaged goods, the second segmentation result includes an instance segmentation map and a product category, and the verification of the second segmentation result based on the weight change includes:

[0020] Based on the instance segmentation map, the product category, and the preset product weight, the total product weight is calculated.

[0021] Determine whether the weight change matches the total weight of the goods;

[0022] If so, then the verification is successful.

[0023] Preferably, in the segmentation method for the image of the flexible packaged goods, the step of extracting features based on the preprocessed image to obtain first texture features, contour features, and watershed boundary features includes:

[0024] Based on the preprocessed image and the Tamura algorithm, feature extraction is performed to obtain the first texture feature;

[0025] Based on the preprocessed image and the multi-level edge detection algorithm, feature extraction is performed to obtain contour features;

[0026] Based on the preprocessed image and the watershed algorithm with label control, feature extraction is performed to obtain the watershed boundary features.

[0027] Preferably, in the method for segmenting images of flexible packaged goods, determining whether the preprocessed image is an image of the same type of goods or an image of different types of goods includes:

[0028] Feature extraction is performed on the preprocessed image to obtain color features, shape features, and a second texture feature;

[0029] Based on the color features, the shape features, the second texture features, and the preset product feature library, the color feature similarity, shape feature similarity, and texture feature similarity are calculated respectively.

[0030] The total similarity is obtained by weighting the color feature similarity, the shape feature similarity, and the texture feature similarity.

[0031] Determine whether the total similarity is greater than or equal to a preset similarity threshold.

[0032] If so, then the preprocessed image is confirmed to be an image of the same type of product;

[0033] If not, then the preprocessed image is confirmed to be an image of a different type of product.

[0034] Preferably, in the method for segmenting the image of a flexible packaged product, the step of extracting features from the preprocessed image to obtain color features, shape features, and a second texture feature includes:

[0035] Based on the preprocessed image, the global mean of the three HSV channels is calculated to obtain color features;

[0036] Based on the preprocessed image, Hu moments are calculated to obtain shape features;

[0037] Based on the preprocessed image, the gray-level co-occurrence matrix is ​​calculated to obtain the second texture feature.

[0038] Preferably, in the segmentation method for the image of soft-packaged goods, the semantic segmentation model adopts the U-Net model, and the instance segmentation model adopts the Mask R-CNN model.

[0039] The second objective of this application is to provide a segmentation system for images of flexible packaged goods.

[0040] The second objective of this application is achieved through the following technical solution:

[0041] A segmentation system for images of soft-packaged goods, applied to self-service vending machines, includes:

[0042] The acquisition unit is used to acquire real-time images of soft-packaged goods at the moment they are picked up, and to preprocess the real-time images to obtain preprocessed images.

[0043] The extraction unit is used to extract features based on the preprocessed image to obtain first texture features, contour features and watershed boundary features;

[0044] The fusion unit is used to perform feature fusion based on the first texture feature, the contour feature, and the watershed boundary feature to obtain fused features;

[0045] The judgment unit is used to determine whether the preprocessed image is an image of the same type of product or an image of a different type of product;

[0046] The first segmentation unit is used to input the fused features into a pre-trained semantic segmentation model to obtain a first segmentation result if it is determined that the preprocessed image is a product image of the same type.

[0047] The second segmentation unit is used to input the fusion features into a pre-trained instance segmentation model to obtain a second segmentation result if the preprocessed image is determined to be an image of a different type of product.

[0048] The verification unit is used to obtain the weight change of the shelves in the self-service vending machine before and after the soft-packaged goods are taken out, and to verify the first segmentation result or the second segmentation result based on the weight change. If the verification is successful, the first segmentation result or the second segmentation result is output.

[0049] The third objective of this application is to provide a segmentation terminal for images of flexible packaged goods.

[0050] The aforementioned objective three of this application is achieved through the following technical solution:

[0051] A segmentation terminal for images of flexible packaged goods includes: a storage medium and a processor;

[0052] The storage medium stores computer-executed instructions.

[0053] The processor executes computer execution instructions stored in the storage medium to implement the segmentation method for flexible packaged product images as described above.

[0054] The fourth objective of this application is to provide a computer-readable storage medium.

[0055] The fourth objective of this application is achieved through the following technical solution:

[0056] A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the segmentation method for flexible packaged product images as described above.

[0057] The above technical solution, applied to self-service vending machines, firstly acquires real-time images of soft-packaged goods at the moment they are taken, and preprocesses these images to obtain preprocessed images. Based on the preprocessed images, feature extraction is performed to obtain first texture features, contour features, and watershed boundary features. Based on the first texture features, contour features, and watershed boundary features, feature fusion is performed to obtain fused features. These fused features are used as input to the subsequent segmentation model, enabling the segmentation model to accurately locate target edges through watershed boundaries, constrain target shapes through contour features, and distinguish regional attributes through the first texture features. This effectively addresses the shortcomings of single-dimensional features and improves segmentation accuracy.

[0058] Then, it is determined whether the preprocessed image is a similar product image or a dissimilar product image. If the preprocessed image is a similar product image, the features are fused and input into the pre-trained semantic segmentation model to obtain the first segmentation result. If the preprocessed image is a dissimilar product image, the features are fused and input into the pre-trained instance segmentation model to obtain the second segmentation result. This is to perform instance-level segmentation on dissimilar product images and semantic fine-grained segmentation on similar product images, thereby improving the segmentation effect.

[0059] Finally, obtain the weight change of the shelves in the self-service vending machine before and after the soft-packaged goods are taken out; based on the weight change, verify the first segmentation result or the second segmentation result. If the verification is successful, output the first segmentation result or the second segmentation result. In this way, the segmentation result is verified by physical gravity data to ensure that the segmentation result conforms to the actual quantity / weight logic of the goods.

[0060] In summary, the above technical solutions can improve the segmentation accuracy of adhesive images of multi-soft packaged goods. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart illustrating a method for segmenting an image of a soft-packaged product according to an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of the structure of a segmentation system for images of flexible packaged goods in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of the structure of a segmentation terminal for a flexible packaged product image in an embodiment of this application. Detailed Implementation

[0065] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described below are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, and can be electrical, mechanical, or other forms.

[0067] It should be understood that the use of terms such as "system," "device," "unit," and / or "module" in this application is merely one method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0068] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.

[0069] If a flowchart is used in this application, it is used to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0070] It should also be noted that, in this document, terms such as “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the article or device that includes the aforementioned element.

[0071] The embodiments in this application are written in a progressive manner.

[0072] like Figure 1 As shown, this application embodiment provides a method for segmenting images of soft-packaged goods, applied to self-service vending machines, including:

[0073] S101. Acquire a real-time image of the soft-packaged product at the moment it is picked up, and preprocess the real-time image to obtain a preprocessed image;

[0074] Specifically, in S101, a high-definition camera built into the self-service vending machine can capture images of the soft-packaged goods the moment a user takes them, thus obtaining a real-time image of the soft-packaged goods at the moment they are taken. Users may take multiple soft-packaged goods together, and the real-time image may show multiple soft-packaged goods glued together. These multiple soft-packaged goods may be of the same type (same category, same packaging) or dissimilar types (different categories, different packaging). Therefore, accurate image segmentation is required for subsequent product identification and checkout.

[0075] To improve image quality, the acquired real-time images also need to be preprocessed to obtain preprocessed images. Existing image preprocessing algorithms can be used to preprocess the real-time images. For example, in some embodiments, preprocessing may include one or more of image cropping, scaling, image denoising, standardization, and normalization, thereby reducing environmental areas unrelated to the product in the image, eliminating noise interference and scale interference. This application is not limited to these methods.

[0076] S102. Based on the preprocessed image, perform feature extraction to obtain the first texture feature, contour feature and watershed boundary feature;

[0077] In step S102, feature extraction is a key step in improving the segmentation accuracy of adhesive images of flexible packaging products. By extracting the first texture feature from the preprocessed image, the segmentation model can distinguish products with similar visual edges but different textures. By extracting contour features from the preprocessed image, the shape rationality of the segmentation result can be constrained, preventing adhesive products from being segmented into irregular shapes. By extracting watershed boundary features from the preprocessed image, the segmentation model can accurately locate the pixel-level separation edges of the adhesive products, solving the problem of blurred edges at the adhesion points. These three types of features complement each other from the three dimensions of "edge localization, shape constraint, and region differentiation," effectively addressing the limitations of single features in the segmentation of adhesive images of flexible packaging products.

[0078] In some embodiments, one implementation of this step specifically includes:

[0079] S1021. Based on the preprocessed image and the Tamura algorithm, feature extraction is performed to obtain the first texture feature;

[0080] Specifically, in S1021, the Tamura algorithm (Tamura features) is a set of methods for describing image texture features. Based on the human visual system's perception of texture, this method quantifies texture in six dimensions: coarseness, contrast, directionality, linelikeness, regularity, and roughness. Preprocessed images can be used to extract features based on the Tamura algorithm, obtaining these six texture features as the first texture feature. This accurately reflects the texture differences in product packaging, helping the segmentation model effectively distinguish between similar / dissimilar adhered products of different materials. Furthermore, Tamura features do not rely on large amounts of labeled data or model training, making them suitable for small sample sizes or resource-constrained scenarios. They are computationally efficient, insensitive to changes in lighting conditions in the shelving environment, and can be fused with other features to improve segmentation accuracy.

[0081] S1022. Based on the preprocessed image and multi-level edge detection algorithm, feature extraction is performed to obtain contour features;

[0082] In step S1022, specifically, a multi-level edge detection algorithm is used to extract features from the region contours of the preprocessed image, resulting in contour features. This multi-level edge detection algorithm encompasses a series of image processing algorithms, including grayscale conversion, Gaussian filtering, histogram equalization, and the Canny edge detection operator. In practical applications, grayscale conversion, Gaussian filtering, and histogram equalization can be performed on the image first, followed by the Canny operator to extract contour features. This effectively captures the contour details of flexible packaging products, providing accurate shape constraints for subsequent feature fusion construction.

[0083] S1023. Based on the preprocessed image and the watershed algorithm with label control, feature extraction is performed to obtain the watershed boundary features.

[0084] Specifically, in S1023, the Marker-Controlled Watershed Algorithm is an improved watershed segmentation method designed to overcome the oversegmentation problem caused by excessive image noise or local minima in gradients in traditional watershed algorithms. It involves preprocessing the image, calculating the gradient image, generating foreground and background markers, merging the markers to construct a marker image, and then performing a marker-controlled watershed transformation on the gradient image to extract watershed boundary features. The watershed boundary is located at the point of maximum gradient, which helps the segmented contour conform to the actual shape of the object and accurately locates the boundaries of the bonded goods. The generated watershed boundary features can be combined with contour features and first texture features, significantly improving the segmentation accuracy of the segmentation model for bonded goods.

[0085] S103. Based on the first texture feature, contour feature and watershed boundary feature, feature fusion is performed to obtain the fused feature;

[0086] In S103, specifically, existing linear fusion technology can be used to fuse the first texture feature, contour feature, and watershed boundary feature to obtain fused features. The fused features are used as input to the subsequent segmentation model, enabling the segmentation model to accurately locate the target edge through the watershed boundary, constrain the target shape through the contour feature, and distinguish regional attributes through the first texture feature. This effectively solves the shortcomings of single-dimensional features and helps improve segmentation accuracy.

[0087] S104. Determine whether the preprocessed image is a similar product image or a dissimilar product image. If the preprocessed image is a similar product image, execute S105. If the preprocessed image is a dissimilar product image, execute S106.

[0088] In step S104, specifically, the preprocessed image can be classified to distinguish between images of similar (same category, same packaging) and images of dissimilar (different categories, different packaging) products. If the images are of similar product type, step S105 is performed for subsequent semantic segmentation; if the images are of dissimilar product type, step S106 is performed for subsequent instance segmentation. In some embodiments, a pre-trained image classification model can be used to classify the preprocessed image to determine whether it is an image of similar or dissimilar products. The image classification model can be based on an existing lightweight image classification network and trained using pre-labeled image samples of similar and dissimilar product categories. Other reasonable methods can also be used to determine whether the preprocessed image is an image of similar or dissimilar products; this application is not limited to these methods.

[0089] S105. Input the fused features into the pre-trained semantic segmentation model to obtain the first segmentation result;

[0090] In S105, specifically, if it is determined that the preprocessed image is an image of the same type of product, the fused features are input into the pre-trained semantic segmentation model to obtain the first segmentation result. The first segmentation result may include a pixel-level semantic segmentation map corresponding to each type of soft-packaged product. Based on the segmentation map, the number of the same type of product can also be calculated. The semantic segmentation map can accurately mark the pixel region of each sticky type of soft-packaged product in the preprocessed image. Even if multiple types of products are in close contact, the independent regions of each product can be clearly divided.

[0091] In some embodiments, the semantic segmentation model employs the U-Net model. U-Net is a fully convolutional neural network (FCN) with an encoder-decoder structure, particularly suitable for small-sample, pixel-level fine-grained segmentation scenarios (such as segmentation of stuck-together products). The U-Net structure consists of three parts: an encoder, a bottleneck layer, and a decoder. The encoder downsamples to extract deep features, the bottleneck layer fuses the deep semantic features transmitted by the encoder, and the decoder upsamples to restore resolution. Through skip connections, shallow features from the corresponding layer of the encoder (such as product edges, contours, and gaps between adhesive segments) are directly concatenated to the feature map of the same layer in the decoder, compensating for the loss of details during the upsampling process. Finally, the output layer maps the feature map to pixel-level segmentation results. It can collect multi-scene image samples (including different bonding strengths and locations) of 2-5 (similar) soft-packaged goods bonded together to construct a small-sample bonding dataset. Then, based on the preprocessing, feature extraction, and feature fusion methods in S101, S102, and S103, fused feature samples are obtained. Then, using existing model training methods, the initial U-Net model is trained using the fused feature samples to obtain a trained semantic segmentation model. Pixel-level annotation of bonded product images is costly and samples are difficult to obtain, while U-Net can achieve high segmentation accuracy without massive amounts of data, significantly reducing annotation costs in practical applications. In addition, U-Net supports end-to-end training, eliminating the need for complex intermediate steps. It directly inputs fused features and outputs pixel-level segmentation masks, seamlessly integrating with feature fusion frameworks and can be deployed on edge devices to meet real-time segmentation requirements.

[0092] S106. Input the fused features into the pre-trained instance segmentation model to obtain the second segmentation result;

[0093] In step S106, specifically, if the preprocessed image is determined to be an image of out-of-category products, the fused features are input into a pre-trained instance segmentation model to obtain a second segmentation result. This second segmentation result can include an instance segmentation map and product categories, such as the instance segmentation mask for each out-of-category soft-packaged product, the bounding box coordinates of each product, and the product category label. Based on this segmentation result, the number of products of each product category in the out-of-category product image can be further counted. The instance segmentation map can accurately distinguish between different categories of bonded products in the image, even if out-of-category products overlap or have tightly joined edges, enabling independent region division and category labeling for each product instance.

[0094] In some embodiments, the instance segmentation model employs the Mask R-CNN model. Mask R-CNN is an instance segmentation framework based on Faster R-CNN, capable of identifying the category and location (boundary) of each object in an image, and outputting a pixel-level segmentation mask for each object. Its core structure includes a feature extraction backbone network, a Region Proposal Network (RPN), a Region of Interest Alignment (RoIAlign) module, a category prediction branch, a bounding box regression branch, and a mask prediction branch. After extracting image features through the feature extraction backbone network, the RPN generates candidate regions; RoIAlign maps the candidate regions onto the feature map while maintaining spatial resolution; the category prediction branch outputs the product category for each candidate region, the bounding box regression branch optimizes the position of the candidate region, and the mask prediction branch generates a binary mask for each candidate region, achieving pixel-level instance segmentation. This method can collect multi-scene image samples (including different adhesion strengths and locations) of 2-5 (different types) of soft-packaged goods bonded together. For each sample, the product instance is labeled with a category label, bounding box, and pixel-level mask to construct a small-sample bonding dataset. Then, based on the preprocessing, feature extraction, and feature fusion methods in S101, S102, and S103, fused feature samples are obtained. Next, existing model training methods are used to train the initial Mask R-CNN model using the fused feature samples, resulting in a trained instance segmentation model capable of simultaneously detecting, classifying, and segmenting product instances. The Mask R-CNN model possesses instance-level segmentation capabilities, directly distinguishing bonded product individuals, supporting multi-class segmentation, and is well-suited for diverse product shelves and subsequent product counting and weight calculation needs.

[0095] S107. Obtain the weight change of the shelves in the self-service vending machine before and after the soft-packaged goods are taken out. Based on the weight change, verify the first segmentation result or the second segmentation result. If the verification is successful, output the first segmentation result or the second segmentation result.

[0096] In S107, specifically, after obtaining the first or second segmentation result, to further avoid misjudging a single product, physical gravity data is introduced to quantify and verify the visual segmentation result. The weight sensor built into the vending machine can be used to obtain the weight change of the soft-packaged product before and after it is removed from the shelf. This weight change is then combined with the product quantity and / or product category statistics from the segmentation result, and matched with preset product weight data to verify the first or second segmentation result. When the verification is successful, the corresponding first or second segmentation result is output. This fusion verification of visual segmentation results and physical weight data forms a dual-protection closed loop of "image semantic / instance segmentation + weight quantization verification," effectively avoiding segmentation misjudgments in scenarios where soft-packaged products are adhered, ensuring the accuracy of product recognition and settlement in the vending machine, and improving the smoothness and trust of users' self-service shopping.

[0097] In some embodiments, one implementation of the step of verifying the first segmentation result based on the weight change specifically includes:

[0098] S1071. Based on the first segmentation result, determine the quantity of goods;

[0099] S1072 calculates the weight of a single item based on the weight change and the quantity of goods, and determines whether the weight of a single item matches the preset product weight database. If so, the verification is confirmed to be successful.

[0100] Specifically, the first segmentation result can include a pixel-level semantic segmentation map corresponding to each type of soft-packaged product. Based on this segmentation map, the quantity N of products of the same type can be calculated. Then, the weight change G is divided by the quantity N to obtain the weight of a single product W = G / N. A preset product weight database stores the standard weight S of various products. To determine whether the weight W of a single product matches the preset product weight database, the absolute value of the difference |WS| between the weight W of a single product and the standard weight S of various products in the preset product weight database is calculated. It is then determined whether there is a case where the absolute value of the difference |WS| is less than or equal to a first preset threshold P. If so, the verification is confirmed to be successful. The first preset threshold P can be set based on the requirements. For example, assuming the first preset threshold P is 2g, when the calculated weight of a single product is 501g, the absolute value of the difference between it and the standard weight 500g of a certain type of product in the preset product weight database is 1g, which is less than 2g. In this case, the verification is confirmed to be successful. In other embodiments, if there is no case where the absolute value of the difference |WS| is less than or equal to the first preset threshold P, the verification is confirmed to have failed. In this case, the step of re-executing the fused features, inputting them into the pre-trained semantic segmentation model, and obtaining the first segmentation result can be returned, or the fused features can be uploaded to the cloud for secondary verification. This application does not impose specific restrictions on this.

[0101] In other embodiments, the second segmentation result includes an instance segmentation map and product categories. One implementation of the step of verifying the second segmentation result based on weight variation specifically includes:

[0102] S1073. Calculate the total weight of the goods based on the instance segmentation map, product category, and preset product weight;

[0103] S1074. Determine whether the weight change matches the total weight of the goods. If so, the verification is confirmed to be successful.

[0104] Specifically, the second segmentation result may include an instance segmentation map and product categories. Based on this segmentation result, the number of products in each product category in the outlier product image can be further counted. The preset product weight includes the standard weight S of each product category. Based on the above data, the total product weight L can be calculated. It is then determined whether the weight change G matches the total product weight L. Specifically, the absolute value of the difference between the weight change G and the total product weight L, |GL|, is calculated. It is then determined whether the absolute value of the difference |GL| is less than or equal to a second preset threshold Q. If so, the verification is confirmed to be successful. The second preset threshold Q can be set according to the requirements. In some other embodiments, if the absolute value of the difference |WS| is greater than the second preset threshold Q, the verification is confirmed to be unsuccessful. In this case, the process can be repeated by inputting the fused features into the pre-trained instance segmentation model to obtain the second segmentation result, or the fused features can be uploaded to the cloud for secondary verification. This application does not impose specific restrictions on this.

[0105] Existing smart self-service vending machines allow products to be freely placed inside. A built-in high-definition camera accurately scans all items, and product recognition is achieved through image processing and visual algorithms. However, when multiple soft-packaged items are taken together, they are prone to sticking together due to static electricity or compression. Traditional image segmentation algorithms, relying on only a single feature, cannot accurately segment the adhesive boundaries, easily misclassifying them as a single item, leading to a high checkout error rate.

[0106] The above embodiment is applied to a self-service vending machine. First, a real-time image of the soft-packaged goods is captured at the moment they are taken, and the real-time image is preprocessed to obtain a preprocessed image. Based on the preprocessed image, feature extraction is performed to obtain a first texture feature, contour feature, and watershed boundary feature. Based on the first texture feature, contour feature, and watershed boundary feature, feature fusion is performed to obtain a fused feature. The fused feature is used as input to the subsequent segmentation model, so that the segmentation model can accurately locate the target edge through the watershed boundary, constrain the target shape through the contour feature, and distinguish the regional attributes through the first texture feature. This effectively solves the shortcomings of single-dimensional features and helps to improve the segmentation accuracy.

[0107] Then, it is determined whether the preprocessed image is a similar product image or a dissimilar product image. If the preprocessed image is a similar product image, the features are fused and input into the pre-trained semantic segmentation model to obtain the first segmentation result. If the preprocessed image is a dissimilar product image, the features are fused and input into the pre-trained instance segmentation model to obtain the second segmentation result. This is to perform instance-level segmentation on dissimilar product images and semantic fine-grained segmentation on similar product images, thereby improving the segmentation effect.

[0108] Finally, obtain the weight change of the shelf in the self-service vending machine before and after the soft-packaged goods are taken out; based on the weight change, verify the first segmentation result or the second segmentation result. If the verification is successful, output the first segmentation result or the second segmentation result. In this way, the segmentation result is verified by physical gravity data to ensure that the segmentation result conforms to the actual quantity / weight logic of the goods.

[0109] In summary, the above embodiments can improve the segmentation accuracy of adhesive images of multi-soft packaged goods.

[0110] In other embodiments of this application, one implementation of the step of determining whether a preprocessed image is a similar product image or a dissimilar product image specifically includes:

[0111] S201. Perform feature extraction on the preprocessed image to obtain color features, shape features, and second texture features;

[0112] In step S201, specifically, based on the preprocessed image, the global mean of the three HSV channels is calculated to obtain color features. For example, after converting the preprocessed image to the HSV color space, the pixel average values ​​of its H (hue), S (saturation), and V (brightness) channels across the entire image are calculated to obtain a three-dimensional vector as a color feature, thereby extracting the main color tone features of the product. Based on the preprocessed image, Hu moments are calculated to obtain shape features; where Hu moments are a set of seven image moment features that are invariant to translation, rotation, and scale, used to characterize the global shape features of the product. Based on the preprocessed image, the gray-level co-occurrence matrix is ​​calculated to obtain the second texture features. The gray-level co-occurrence matrix (GLCM) characterizes the structural, directional, and uniform properties of texture by statistically analyzing the frequency of gray-level values ​​co-occurring in pixel pairs under specific spatial relationships in the image, which can help distinguish the packaging materials / patterns of different product categories.

[0113] S202. Based on color features, shape features, second texture features, and a preset product feature library, calculate the color feature similarity, shape feature similarity, and texture feature similarity respectively;

[0114] In S202, specifically, the preset product feature library stores the benchmark color features, benchmark shape features, and benchmark second texture features of various products; by calculating the Euclidean distance between the color features of the preprocessed image and the benchmark color features of various products and converting it into a similarity, the color feature similarity between the preprocessed image and various products is obtained; by calculating the Euclidean distance between the shape features of the preprocessed image and the benchmark shape features of various products and converting it into a similarity, the color feature similarity between the preprocessed image and various products is obtained; by calculating the cosine similarity between the second texture feature of the preprocessed image and the benchmark second texture features of various products, the texture feature similarity between the preprocessed image and various products is obtained.

[0115] S203. Based on the color feature similarity, shape feature similarity, and texture feature similarity, perform weighted calculation to obtain the total similarity;

[0116] In S203, specifically, based on the above three feature similarities between the preprocessed image and various products, the following formula is used for weighted calculation to obtain the total similarity between the preprocessed image and various products;

[0117] Stotal = α × Scolor + β × Sshape + γ × Stexture;

[0118] In the formula, Stotal represents the total similarity, Scolor represents the color feature similarity, Sshape represents the shape feature similarity, Stexture represents the texture feature similarity, and α, β, and γ represent preset weights, and their specific values can be set based on actual needs. For example, the color weight α = 0.4, the shape weight β = 0.3, and the texture weight γ = 0.3. This application does not make specific restrictions on this.

[0119] S204. Determine whether the total similarity is greater than or equal to the preset similarity threshold. If so, confirm that the preprocessed image is a homogeneous product image; if not, confirm that the preprocessed image is a heterogeneous product image.

[0120] In S204, specifically, a similarity threshold T can be set in advance, and the size of the total similarity between the preprocessed image and various products and the preset similarity threshold T is compared. If Stotal ≥ T, confirm that the preprocessed image is a homogeneous product image; if Stotal < T, confirm that the preprocessed image is a heterogeneous product image.

[0121] In this embodiment, it is possible to achieve the classification and judgment of the preprocessed image without training an image classification model, and it has low requirements for hardware computing power. In addition, the preset product feature library is easy to maintain. For new product categories, only the benchmark features need to be supplemented, and there is no need to retrain the model. The calculation is efficient, the implementation cost is low, and the judgment result is interpretable, which can better adapt to the rapid classification requirements of product images.

[0122] like Figure 2 As shown, in another embodiment of this application, a segmentation system for images of soft-packaged goods is provided, applied to a self-service vending machine, comprising:

[0123] The acquisition unit 10 is used to acquire real-time images of soft-packaged goods at the moment they are picked up, and to preprocess the real-time images to obtain preprocessed images.

[0124] Extraction unit 11 is used to extract features based on the preprocessed image to obtain first texture features, contour features and watershed boundary features;

[0125] The fusion unit 12 is used to perform feature fusion based on the first texture feature, contour feature and watershed boundary feature to obtain fused features;

[0126] Judgment unit 13 is used to determine whether the preprocessed image is an image of the same type of product or an image of a different type of product;

[0127] The first segmentation unit 14 is used to input the fused features into the pre-trained semantic segmentation model to obtain the first segmentation result if it is determined that the pre-processed image is a product image of the same type.

[0128] The second segmentation unit 15 is used to input the fused features into the pre-trained instance segmentation model if it is determined that the pre-processed image is an image of a different type of product, so as to obtain the second segmentation result.

[0129] The verification unit 16 is used to obtain the weight change of the shelf in the self-service vending machine before and after the soft packaged goods are taken out. Based on the weight change, the first segmentation result or the second segmentation result is verified. If the verification is successful, the first segmentation result or the second segmentation result is output.

[0130] like Figure 3 As shown, in another embodiment of this application, a segmentation terminal for images of soft-packaged goods is also provided, including: a storage medium 20 and a processor 21;

[0131] Storage medium 20 stores computer-executed instructions;

[0132] The processor 21 executes computer execution instructions stored in the storage medium 20 to implement the segmentation method for the flexible packaged product image as described above.

[0133] The processor 21 may include one or more processing cores. The processor 21 executes instructions, programs, code sets, or instruction sets stored in the storage medium 20, and calls data stored in the storage medium 20 to perform various functions and process data as described in this application. The processor 21 may be at least one of a specific application-specific integrated circuit, digital signal processor, digital signal processing device, programmable logic device, field-programmable gate array, central processing unit, controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the functions of the processor 21 may also be other types.

[0134] The storage medium 20 can be used to store instructions, programs, code, code sets, or instruction sets. The storage medium 20 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the aforementioned segmentation method for the flexible packaged product image; the data storage area may store data involved in the aforementioned segmentation method for the flexible packaged product image.

[0135] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions that, when executed by a processor, are used to implement the segmentation method for flexible packaged product images as described above.

[0136] The computer-readable storage medium can be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory, random access memory, or optical disc.

[0137] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for segmenting images of flexible packaged goods, characterized in that, Applications to self-service vending machines include: Real-time images of soft-packaged goods are captured at the moment they are picked up, and the real-time images are preprocessed to obtain preprocessed images; Based on the preprocessed image, feature extraction is performed to obtain the first texture feature, contour feature, and watershed boundary feature; Based on the first texture feature, the contour feature, and the watershed boundary feature, feature fusion is performed to obtain the fused feature; Determine whether the preprocessed image is an image of the same type of product or an image of a different type of product; If the preprocessed image is a product image of the same type, then the fused features are input into the pre-trained semantic segmentation model to obtain the first segmentation result; If the preprocessed image is an image of a different type of product, then the fused features are input into a pre-trained instance segmentation model to obtain a second segmentation result; Obtain the weight change of the shelves in the self-service vending machine before and after the soft-packaged goods are taken out. Based on the weight change, verify the first segmentation result or the second segmentation result. If the verification is successful, output the first segmentation result or the second segmentation result.

2. The method as described in claim 1, characterized in that, The verification of the first segmentation result based on the weight change includes: Based on the first segmentation result, determine the quantity of goods; Based on the weight change and the quantity of goods, the weight of a single item is calculated, and it is determined whether the weight of a single item matches a preset product weight database. If so, then the verification is successful.

3. The method as described in claim 1, characterized in that, The second segmentation result includes an instance segmentation map and product categories. The verification of the second segmentation result based on the weight change includes: Based on the instance segmentation map, the product category, and the preset product weight, the total product weight is calculated. Determine whether the weight change matches the total weight of the goods; If so, then the verification is successful.

4. The method as described in claim 1, characterized in that, Based on the preprocessed image, feature extraction is performed to obtain first texture features, contour features, and watershed boundary features, including: Based on the preprocessed image and the Tamura algorithm, feature extraction is performed to obtain the first texture feature; Based on the preprocessed image and the multi-level edge detection algorithm, feature extraction is performed to obtain contour features; Based on the preprocessed image and the watershed algorithm with label control, feature extraction is performed to obtain the watershed boundary features.

5. The method as described in claim 1, characterized in that, The step of determining whether the preprocessed image is an image of the same type of product or an image of a different type of product includes: Feature extraction is performed on the preprocessed image to obtain color features, shape features, and a second texture feature; Based on the color features, the shape features, the second texture features, and the preset product feature library, the color feature similarity, shape feature similarity, and texture feature similarity are calculated respectively. The total similarity is obtained by weighting the color feature similarity, the shape feature similarity, and the texture feature similarity. Determine whether the total similarity is greater than or equal to a preset similarity threshold. If so, then the preprocessed image is confirmed to be an image of the same type of product; If not, then the preprocessed image is confirmed to be an image of a different type of product.

6. The method as described in claim 5, characterized in that, The step of extracting features from the preprocessed image to obtain color features, shape features, and a second texture feature includes: Based on the preprocessed image, the global mean of the three HSV channels is calculated to obtain color features; Based on the preprocessed image, Hu moments are calculated to obtain shape features; Based on the preprocessed image, the gray-level co-occurrence matrix is ​​calculated to obtain the second texture feature.

7. The method as described in claim 1, characterized in that, The semantic segmentation model uses the U-Net model, and the instance segmentation model uses the Mask R-CNN model.

8. A segmentation system for images of flexible packaged goods, characterized in that, Applications to self-service vending machines include: The acquisition unit is used to acquire real-time images of soft-packaged goods at the moment they are picked up, and to preprocess the real-time images to obtain preprocessed images. The extraction unit is used to extract features based on the preprocessed image to obtain first texture features, contour features and watershed boundary features; The fusion unit is used to perform feature fusion based on the first texture feature, the contour feature, and the watershed boundary feature to obtain fused features; The judgment unit is used to determine whether the preprocessed image is an image of the same type of product or an image of a different type of product; The first segmentation unit is used to input the fused features into a pre-trained semantic segmentation model to obtain a first segmentation result if it is determined that the preprocessed image is a product image of the same type. The second segmentation unit is used to input the fusion features into a pre-trained instance segmentation model to obtain a second segmentation result if the preprocessed image is determined to be an image of a different type of product. The verification unit is used to obtain the weight change of the shelves in the self-service vending machine before and after the soft-packaged goods are taken out, and to verify the first segmentation result or the second segmentation result based on the weight change. If the verification is successful, the first segmentation result or the second segmentation result is output.

9. A segmentation terminal for images of flexible packaged goods, characterized in that, include: Storage media and processor; The storage medium stores computer-executed instructions. The processor executes computer execution instructions stored in the storage medium to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images