Image-based Product Aggregation Method, Device, and Electronic Equipment

Abstract

The present application discloses an image-based product aggregation method, belonging to the field of computer technology, which is used to solve the problem of inaccurate product aggregation results caused by simply comparing the feature distance and threshold in the prior art for image-based product aggregation. The product aggregation method disclosed in the present application includes: determining the image regions and image features corresponding to each product included in the target image; aggregating the image regions corresponding to the products according to the image features corresponding to the products, and determining the product regions corresponding to the products; retrieving the product images of each product in a preset standard item database to determine the item retrieval results of each product; aggregating the product regions corresponding to the products according to the item retrieval results to determine the aggregation results of each product included in the target image. The present application corrects the preliminary aggregation results through the retrieval results, improving the accuracy of the product aggregation results.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to an image-based product aggregation method, apparatus, and electronic device. Background Art

[0002] In a retail application scenario, aggregating products on a shelf to determine the inventory information of the products is an important means for applications such as inventory monitoring or management. In the prior art, when aggregating shelf products, typically, an image is collected, and then, by detecting the shelf image, the image of each product on the shelf is determined; afterwards, based on the product features extracted from the image of each product, by calculating the distance between different product features and comparing it with a certain threshold T, the products are aggregated. Since the product aggregation method in the prior art needs to be compared with a specific threshold, the determination of this specific threshold directly affects the accuracy of the aggregation result. However, in the case where there are a large number of shelf product types and there are environmental differences such as light on different shelves, it is difficult to obtain a reasonable threshold. Therefore, when aggregating products on a shelf in the prior art, there is at least a defect that the aggregation result is inaccurate. Summary of the Invention

[0003] An embodiment of this application provides an image-based product aggregation method, which helps to improve the accuracy of product aggregation.

[0004] To solve the above problems, in a first aspect, an embodiment of this application provides an image-based product aggregation method, including:

[0005] Determine the image regions and image features corresponding to each product included in the target image;

[0006] Aggregate the image regions corresponding to the products according to the image features corresponding to each product, and determine the product regions corresponding to each product;

[0007] Retrieve the product images of each product in a preset standard item database to determine the item retrieval results of each product, where the product image is a part of the target image covered by the image region corresponding to the product;

[0008] Aggregate the product regions corresponding to each product according to the item retrieval results to determine the aggregation results of each product included in the target image.

[0009] In a second aspect, an embodiment of this application provides an image-based product aggregation apparatus, including:

[0010] A product image and feature determination module, configured to determine the image regions and image features corresponding to each product included in the target image;

[0011] A product area determination module, configured to aggregate the image areas corresponding to the products according to the image features corresponding to the respective products, and determine the product areas corresponding to the respective products;

[0012] A retrieval module, configured to retrieve the product images of the respective products in a preset standard item database, and determine the item retrieval results of the respective products, where the product image is a part of the target image covered by the image area corresponding to the product;

[0013] An area aggregation module, configured to aggregate the product areas corresponding to the respective products according to the item retrieval results, and determine the aggregation results of the respective products included in the target image.

[0014] In a third aspect, an embodiment of the present application further discloses an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for aggregating products based on images according to the embodiments of the present application is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the method for aggregating products based on images disclosed in the embodiments of the present application are implemented.

[0016] The method for aggregating products based on images disclosed in the embodiments of the present application determines the image areas and image features corresponding to the respective products included in the target image; aggregates the image areas corresponding to the products according to the image features corresponding to the respective products, and determines the product areas corresponding to the respective products; retrieves the product images of the respective products in a preset standard item database, and determines the item retrieval results of the respective products, where the product image is a part of the target image covered by the image area corresponding to the product; aggregates the product areas corresponding to the respective products according to the item retrieval results, and determines the aggregation results of the respective products included in the target image, and solves the problem of inaccurate product aggregation results caused by product aggregation based on the comparison results of feature distances and thresholds in the prior art. Since the method for aggregating products based on images disclosed in the embodiments of the present application sets a step of further correcting the aggregation result based on image features through the retrieval result, the dependence on the absolute threshold is reduced, and the accuracy of the product aggregation result can be further improved. Description of the Drawings

[0017] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 is the flowchart of the image-based product aggregation method according to Embodiment 1 of the present application;

[0019] Figure 2 is the flowchart of the image-based product aggregation method according to Embodiment 2 of the present application;

[0020] Figure 3 is a schematic diagram of the target image of a row of shelves in Embodiment 2 of the present application;

[0021] Figure 4 is a schematic diagram of the detection result of the product area of the target image in Embodiment 2 of the present application;

[0022] Figure 5 is one of the schematic diagrams of the preliminary product aggregation in Embodiment 2 of the present application;

[0023] Figure 6 is a schematic diagram of the left-merging result of the products in Embodiment 2 of the present application;

[0024] Figure 7 is a schematic diagram of the right-merging result of the products in Embodiment 2 of the present application;

[0025] Figure 8 is the second of the schematic diagrams of the preliminary product aggregation result in Embodiment 2 of the present application;

[0026] Figure 9 is a schematic diagram of the final image-based product aggregation result in Embodiment 2 of the present application;

[0027] Figure 10 is one of the schematic diagrams of the structure of the image-based product aggregation device according to Embodiment 3 of the present application;

[0028] Figure 11 is the second of the schematic diagrams of the structure of the image-based product aggregation device according to Embodiment 3 of the present application;

[0029] Figure 12 is the third of the schematic diagrams of the structure of the image-based product aggregation device according to Embodiment 3 of the present application. Detailed implementation manners

[0030] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the scope of protection of the present application.

[0031] The image-based product aggregation method disclosed in the embodiments of the present application can be applied to the warehousing industry, or application scenarios such as the retail industry where products are placed and sold based on shelves for product management and inventory monitoring, etc., and can also be applied to other fields for counting the quantity and / or category of products based on images. For the convenience of readers to more easily understand the technical solutions of the present application, in the embodiments of the present application, an example is given of applying the product aggregation method to the retail industry (such as a supermarket) to illustrate the specific technical solutions of the method.

[0032] When aggregating products in the embodiments of the present application, aggregation is performed based on SKU (Stock Keeping Unit, inventory unit). SKU is the basic unit for measuring inventory in and out, which can be in units of pieces, boxes, pallets, etc. Using SKU for inventory in and out measurement is a necessary method for the logistics management of the distribution centers of large chain supermarkets. In the prior art, SKU has been extended to be the abbreviation of the unified product number, and each product corresponds to a unique SKU. For e-commerce, an SKU refers to a product item, and each item has an SKU, which is convenient for e-commerce brands to identify products. In the specific embodiments of the present application, SKU is also referred to as "product item", and the product item is the most fine-grained representative of the product. For example, different flavors of Coca-Cola are different product items.

[0033] Embodiment 1

[0034] An image-based product aggregation method disclosed in this embodiment, as Figure 1 shown, the method includes: Step 110 to Step 140.

[0035] Step 110, determine the image regions and image features corresponding to each product included in the target image.

[0036] The target image described in the embodiments of the present application can be a shelf image of a row of shelves on which the products are placed and displayed collected by an image acquisition device, or an image of a row of regularly placed products. In the following embodiments, for the convenience of readers to understand the technical solutions of the present application, detailed descriptions will be made in combination with the shelf image.

[0037] In specific application scenarios, there may be multiple columns of shelves arranged side by side. When the collected image includes images of multiple columns of shelves, first, through image detection and processing, an image including only one column of shelves is determined to perform product aggregation for one column of shelves. Specifically in implementation, the image gradient of the collected image is calculated, line detection is performed on the gradient map, and erosion and dilation processing are performed on the detection result to obtain the image of each column of shelves.

[0038] Generally, each column of shelves may include one or more layers. Specifically in implementation, the image of each layer of shelves can be determined by performing image detection and processing on the image of each column of shelves. And further, based on the image of each layer of shelves, the image region corresponding to each product included in that layer of shelves and the image features corresponding to each product are determined. When determining the image of a certain layer of a certain column of shelves, the vertical coordinates (such as represented by y1) of all ink labels in the shelf image can be determined through a pre-trained ink label detection model, and the vertical coordinates of the ink labels are sorted from small to large. If the difference between the vertical coordinates of two adjacent ink labels is greater than twice the average height of the ink labels, the number of layers of the shelf is automatically incremented by 1. Thus, the number of layers of the shelf is automatically detected. The basis is that the position deviation of the ink labels on the same shelf layer of the corrected image is not large and will not exceed twice the height of the ink labels. Or, the shelf image corresponding to each layer of shelves is determined according to the preset shelf layer information.

[0039] Specifically in implementation, the image region corresponding to each product included in a single-layer shelf can be determined by pre-training an image detection model, or the image region corresponding to each product included in a single-layer shelf can be determined according to parameters such as the preset slots and height of the target shelf. Specifically in implementation, an image feature extraction model can be pre-trained to extract the image features of the product from the image region corresponding to each product, or the image features of the product (such as extracting the texture features, color features, contour features, etc. of the image) can be extracted from the image region corresponding to each product through an image processing algorithm.

[0040] Step 120, aggregate the image regions corresponding to the products according to the image features corresponding to the products, and determine the product regions corresponding to the products.

[0041] Furthermore, for each layer of shelves, based on the position information of the image regions corresponding to the products determined in the image of that layer of shelves and the image features of the product extracted from that image region, adjacent products are aggregated.

[0042] For example, for two adjacent image regions, if the feature distance between the image features of the products corresponding to the two adjacent image regions is less than a preset distance threshold, it can be considered that the products in the two adjacent image regions are products of the same item, and thus the two adjacent image regions can be aggregated into an image region including two products. If the feature distance between the image features of the product corresponding to another image region adjacent to the aggregated image region and the image features of the products included in the aggregated image region is less than the preset distance threshold, the aggregation of adjacent image regions can be further performed.

[0043] For another example, based on the relative magnitudes of the feature distances between the image feature of a certain product and the image features of two products adjacent to the product, left-merging and right-merging can be performed on the image region corresponding to the product, and products with higher feature similarity can be aggregated into one image region. Finally, further fusion adjustment is performed on the image regions obtained by left-merging and the image regions obtained by right-merging to determine the division result of the image regions of the products included in this layer of the shelf.

[0044] Step 130, retrieve the product images of each of the products in a preset standard item database to determine the item retrieval results of each of the products.

[0045] Wherein, the product image is a part of the target image covered by the image region corresponding to the product.

[0046] Furthermore, for each image region obtained by dividing each layer of shelf image, retrieve the product images of each of the products included in the image region respectively to determine the item of each of the products included in each image region and the matching probability that each product is recognized as the corresponding item.

[0047] When the present application is specifically implemented, a standard item database needs to be set in advance. The standard item database includes standard images of products of each item, and the standard images are indexed by the corresponding items. Specifically in implementation, by respectively matching each product image with the standard images in the preset standard item database, the preset number of standard images with the highest matching degree to each product image can be determined, as well as the matching probabilities between the product image and the preset number of standard images. Then, for the preset number of standard images matched by each product, a set of retrieval results is formed by combining the matching probability of each standard image and the item used to index the standard image, and the preset number of retrieval results of each product can be obtained. For each product, arrange the preset number of retrieval results in descending order of the matching probability, and the item retrieval result of the corresponding product is obtained.

[0048] When separately matching each product image with the standard images in the pre-set standard item database, the image features of the product can be extracted first, and then the image features are matched with the image features of each standard product image in the pre-set standard item database. The similarity is judged based on the feature distance. The higher the similarity, the greater the matching probability.

[0049] Among them, the product image of each product is the target image within the image area corresponding to each product determined in the foregoing step 110.

[0050] Step 140: Aggregate the product areas corresponding to each product according to the item retrieval result, and determine the aggregation result of each product included in the target image.

[0051] Next, according to the retrieval results of each product in each image area obtained after aggregation, such as the items determined in the foregoing steps, and the corresponding matching probabilities of each item, further determine the items matched by each product area obtained after aggregation. Then, further aggregate the adjacent product areas that match the same item to determine the final product aggregation result of the target layer shelf. In the final product aggregation result of one layer of the shelf, the products included in the adjacent product areas correspond to different items.

[0052] Furthermore, according to the size and position information of the product areas obtained from the final aggregation result, the number of products corresponding to the corresponding items included in each product area obtained by aggregation can be further determined.

[0053] The image-based product aggregation method disclosed in the embodiments of the present application solves the problem of inaccurate product aggregation results caused by product aggregation based on the comparison results of feature distances and thresholds in the prior art by determining the image areas and image features corresponding to each product included in the target image; aggregating the image areas corresponding to the products according to the image features corresponding to each product to determine the product areas corresponding to each product; retrieving the product images of each product in the preset standard item database to determine the item retrieval results of each product, where the product image is a part of the target image covered by the image area corresponding to the product; and aggregating the product areas corresponding to each product according to the item retrieval result to determine the aggregation result of each product. Since the image-based product aggregation method disclosed in the embodiments of the present application sets a step of further correcting the aggregation result based on image features through the retrieval result, the dependence on the absolute threshold is reduced, and the accuracy of the product aggregation result can be further improved.

[0054] Embodiment 2

[0055] An image-based product aggregation method disclosed in this embodiment is as Figure 2As shown, the method includes: step 210 to step 250.

[0056] Step 210, training a product detection model and a product feature extraction model.

[0057] First, collect shelf images as sample data and set sample labels to construct training samples for the product detection model.

[0058] In specific implementation, shelf images of a single-row shelf in a supermarket can be collected, and the product image areas in each shelf image are labeled with annotation boxes, where the annotation boxes are used to identify the image areas where the product images corresponding to each product in each shelf image are located. Then, for each shelf image, the coordinates of the upper left corner and the lower right corner of each annotation box are used as the positions of the image areas corresponding to the annotation box, and the positions of the image areas corresponding to all the annotation boxes included in the shelf image are used as the sample labels of the shelf image. Each shelf image with sample labels is used as a training sample and trained using Faster-RCNN (Faster Region with Convolutional Neural Network), and a product detection model is obtained. It can be understood that other deep learning models such as RCNN (Region with Convolutional Neural Network) and Fast-RCNN (Fast Region with Convolutional Neural Network) that can detect and mark targets from images can also be used.

[0059] In some embodiments of the present application, the image areas where each product and each label are located in each shelf image can also be labeled with annotation boxes. Then, for each shelf image, the coordinates of the upper left corner and the lower right corner of each annotation box are used as the positions of the image areas corresponding to the annotation box, and the image type corresponding to the annotation box is set. The image type is used to indicate whether the annotation box contains a product image or a label image. After that, the positions and image types of the image areas corresponding to all the annotation boxes included in the shelf image are used as the sample labels of the shelf image. Each shelf image with sample labels is used as a training sample to train the product detection model.

[0060] Furthermore, the aforementioned labeled product images can be used as sample data, and the item of the product in the product image is used as a sample label to construct training samples for the product feature extraction model. Then, using a classification model as the basic model, the product feature extraction model is trained.

[0061] Step 220, determining the image areas and image features corresponding to each product included in the target image.

[0062] In the embodiments of the present application, the shelf image (i.e., the target image) may be a shelf image of a column of shelves on which the products are placed and displayed, collected by an image acquisition device.

[0063] In a specific application scenario, there may be multiple columns of shelves arranged side by side. When the collected image includes images of multiple columns of shelves, first, through image detection processing, an image including only one column of shelves is determined, as Figure 3 shown, for product aggregation for one column of shelves. The specific implementation of determining the shelf image corresponding to each column of shelves from the shelf image, and determining the image of one layer of shelves from the shelf image corresponding to one column of shelves can be found in Embodiment 1, which will not be elaborated in this embodiment. Figure 3 In the figure, the rectangular frames marked with different numbers represent different products. The rectangular frames marked with the same number represent the same product.

[0064] Generally, each column of shelves may include one or more layers. In some embodiments of the present application, the steps of determining the image regions and image features corresponding to each product included in the target image include: determining the image regions corresponding to each product included in the target image through a preset product detection model; and determining the image features of each of the image regions through a preset product feature extraction model. For example, Figure 3 when the shelf image shown is input into the product detection model, the product detection model will output the coordinates of the image regions of each recognized product.

[0065] Specifically, when training the product detection model, if the sample label in the training sample is the coordinates and image type of the image region, then Figure 3 after the shelf image shown is input into the product detection model, the product detection model will output the coordinates and image type of each recognized image region. Among them, the image type is used to identify whether the image of the image region is a product image or a label image. Then, all the product images included in the shelf image are determined according to the type of the image region output by the product detection model. Figure 3 The image regions recognized after the shelf image shown is input into the product detection model are as Figure 4 shown. Figure 4 Each dashed box in the figure corresponds to an image region.

[0066] Each recognized product image corresponds to a product. By aggregating the coordinates of the image regions of the product images, the image regions corresponding to each layer of shelves in the shelf image can be further determined, as well as the product images corresponding to the products included in each layer of shelves and the coordinates of the image regions corresponding to each product image.

[0067] Further, for each product image included in each layer of shelf images, the product image is input into the product feature extraction model trained in the foregoing steps to determine the image features of each of the product images. The image features of the product image reflect the features of the corresponding product, and thus can be used for product aggregation.

[0068] Step 230: Aggregate the image regions corresponding to the products according to the image features corresponding to the products, and determine the product regions corresponding to the products.

[0069] Further, for each layer of shelves, based on the position information of the image regions corresponding to the products determined in the shelf image of this layer and the image features of the products extracted from the image regions, adjacent products are aggregated.

[0070] In some embodiments of the present application, the step of aggregating the image regions corresponding to the products according to the image features corresponding to the products to determine the product regions corresponding to the products includes: aggregating the image regions corresponding to the products according to a preset feature distance threshold and the distance between the image features of adjacent products in the image regions corresponding to the products, and determining the product regions corresponding to the products. For example, for Figure 4 in the shelf image of the first layer of shelves, if the feature distance between the image features of the products corresponding to two adjacent image regions 410 and 420 is less than the preset distance threshold T, it can be considered that the products in the two adjacent image regions are products of the same item, and thus the two adjacent image regions are aggregated into an image region including two products, such as Figure 5 510 in. If the feature distance between the image features of the product corresponding to another image region 430 adjacent to the aggregated image region 510 and the image features of the products included in the aggregated image region 510 is less than the preset distance threshold, the aggregation of the adjacent image regions 510 and 430 can be further performed. Among them, the feature distance threshold is determined according to the product aggregation accuracy and the acquisition environment of the shelf image.

[0071] In some preferred embodiments of the present application, the step of aggregating the image regions corresponding to the products according to the image features corresponding to the respective products to determine the product regions corresponding to the respective products includes: aggregating the respective products separately through bilateral merging according to the distances between the image features of adjacent products in the image regions corresponding to the respective products, and determining the product regions corresponding to the respective products. The bilateral merging described in the embodiments of the present application includes left merging and right merging. Specifically, when aggregating products by comparing the distance between image features with an absolute threshold, if the absolute threshold is set too high, it may cause a large distance between the image features of the same product due to the environmental light difference in the placement position, not meeting the requirements of the absolute threshold, and thus the product aggregation is inaccurate. Therefore, in this embodiment, the absolute threshold is not used for product aggregation.

[0072] Specifically, the step of aggregating the respective products separately through bilateral merging according to the distances between the image features of adjacent products in the image regions corresponding to the respective products to determine the product regions corresponding to the respective products includes: performing left merging on the respective products according to the distances between the image features of adjacent two products in the respective products to determine a number of first candidate product regions; and performing right merging on the respective products according to the distances between the image features of adjacent two products in the respective products to determine a number of second candidate product regions; aggregating the number of first candidate product regions and the number of second candidate product regions to determine the product regions corresponding to the respective products. Due to the different angles of product placement in the shelf or the light differences at different positions in the shelf, the image features of adjacent products with the same item will be different and be merged into different product regions. Therefore, through bilateral merging (i.e., left merging + right merging), it helps to improve the accuracy of product region determination.

[0073] When performing left merging, first traverse the products on one layer of the shelf from left to right starting from the leftmost product on the layer, and successively judge the distances between the image features of this product and the image features of the products adjacent to its left and right respectively, and determine whether to aggregate this product with the product adjacent to its left or generate a new product region according to the size of the distance.

[0074] Still taking Figure 4Examples of product image recognition results in the image area corresponding to the first layer of shelves in the shown shelf image. For the leftmost product on the target layer of shelves (such as the first layer of shelves, e.g., the product in image area 410), since it has no adjacent product on the left, it is not aggregated, and a new product area (corresponding to image area 410) is generated. Then, traverse the next product to the right (such as the product in image area 420). First, calculate the image feature distance dist_left between the product in image area 420 and its adjacent product on the left (such as the product in image area 410) and the image feature distance dist_right between the product in image area 420 and its adjacent product on the right (such as the product in image area 430); then, judge the magnitudes of the feature distance dist_left between the product and its adjacent product on the left and the feature distance dist_right between the product and its adjacent product on the right. If dist_left > dist_right, the product is not aggregated, and a new product area (corresponding to image area 420) is generated; if dist_left < dist_right, the product (such as the product in image area 420) is aggregated with its adjacent product on the left (such as the product in image area 410). Then, continue to traverse the next product to the right (such as the product in image area 430) until the rightmost product on the current shelf layer (such as the product in image area 440) is traversed. After performing the left-merging process on the products in the target layer of shelves according to the above method, there will be obtained several first candidate product areas 610 to 650 as shown in Figure 6 where each first candidate product area corresponds to a placement area on the shelf, each first candidate product area includes at least one product, and the products included in each first candidate product area have relatively high similarity.

[0075] When performing right-merging, first start from the rightmost product on a layer of shelves and traverse the products on this layer from right to left, successively judge the distances between the image features of this product and the image features of its adjacent products on the left and right respectively, and determine whether to aggregate this product with its adjacent product on the right or generate a new product area according to the magnitudes of the distances.

[0076] Still taking Figure 4An example of the product image recognition result in the image area corresponding to the first layer of shelves in the shown shelf image. For the rightmost product on the target layer of shelves (the product in image area 440), since it has no adjacent product on the right, it is not aggregated, and a new product area (corresponding to image area 440) is generated. Then, traverse the next product to the left (the product in image area 450). First, calculate the image feature distance dist_left between the product in image area 450 and its adjacent product on the left (the product in image area 460) and the image feature distance dist_right between the product in image area 450 and its adjacent product on the right (the product in image area 440) respectively; then, judge the magnitudes of the feature distance dist_left between this product and its adjacent product on the left and the feature distance dist_right between this product and its adjacent product on the left. If dist_left < dist_right, this product is not aggregated, and a new product area (such as corresponding to image area 450) is generated; if dist_left > dist_right, this product (the product in image area 450) is aggregated with its adjacent product on the right (the product in image area 440). Then, continue to traverse the next product to the left (the product in image area 460). This continues until the leftmost product on the current shelf layer (the product in image area 410) is traversed.

[0077] After performing the right-merging process on the products in the target layer of shelves according to the above method, the following will be obtained as Figure 7 shown, several second candidate product areas 710 to 760, each second candidate product area corresponding to a placement area on the shelf, each second candidate product area including at least one product, and the products included in each second candidate product area having relatively high similarity.

[0078] As can be seen from the above description, after performing the left-merging and right-merging processes on the products in each layer of shelves respectively, two aggregation results for the products on the same layer of shelves will be obtained. Next, these two aggregation results need to be further aggregated to determine the corresponding further aggregation results for the products included in each layer of shelves, that is, initially determine the product areas corresponding to the products included in each layer of shelves. If a certain product is aggregated into the first candidate product area and the second candidate product area corresponding to different shelf positions, then the first candidate product area and the second candidate product area are aggregated. For example, such as Figure 6In the first candidate product area 650, the 8th to 10th products from the left (i.e., 3 products 4) are included. In the aggregated result obtained by right merging, the 8th to 9th products from the left (i.e., 2 products 4) are aggregated together, corresponding to the second candidate product area 720. The 8th to 10th products sku10 are not aggregated with other products and are independently distributed in the second candidate product area 710. In the case where the two aggregation results of product 4 are inconsistent, the first candidate product area 650 and the second candidate product area 710 where product 4 is located are aggregated to obtain as Figure 8 shown in the product area 850.

[0079] Step 240, retrieve the product images of each of the products in the preset standard item database to determine the item retrieval results of each of the products.

[0080] Wherein, the product image is a part of the shelf image covered by the image area corresponding to the product, such as Figure 3 310 in is the product image of product 1.

[0081] Furthermore, for each image area obtained by dividing each layer of shelf image (i.e., the target image), retrieve the product images of each product included in the image area respectively to determine the item of each product included in each image area and the matching probability that each product is recognized as the corresponding item. For the specific implementation of retrieving the product images of each of the products in the preset standard item database to determine the item retrieval results of each of the products, refer to Embodiment 1, and this embodiment will not be elaborated here.

[0082] Step 250, aggregate the product areas corresponding to each of the products according to the item retrieval results to determine the aggregation results of each of the products included in the target image.

[0083] Next, according to the retrieval results of each product in each product area obtained after aggregation, such as the items determined in the foregoing steps, and the corresponding matching probabilities of each item, further determine the item matched by each product area obtained after aggregation. Then, further aggregate the adjacent product areas that match the same item to determine the final product aggregation result of the target layer of the shelf. For example, for each product area, respectively determine the item with the highest matching probability in the retrieval results of each product in the product area to construct the candidate item set of the product area, and take the item that recalls the most of the candidate item set as the item corresponding to the products in the product area; or, first calculate the average similarity between each item in the candidate item set and all the products in the product area, and select the item with the highest average similarity as the item corresponding to the product area. Then, aggregate the adjacent product areas that correspond to the same item to determine the aggregation results of each of the products included in the shelf image.

[0084] In some preferred embodiments of the present application, the item retrieval results include: the items corresponding to the products, and the matching probabilities between the products and the corresponding items. The step of aggregating the product regions corresponding to each product according to the item retrieval results to determine the aggregation results of each product included in the target image includes: for each product region, performing item weighted voting based on the matching probabilities according to the item retrieval results of each product in the product region to determine the item corresponding to the product region; aggregating the product regions that are adjacent and correspond to the same item to determine the aggregation results of each product included in the target image. In order to improve the accuracy of the aggregation results, more recalled items and the corresponding matching probabilities are selected to determine the items corresponding to each product region.

[0085] For example, first, for each product region, the item with the highest matching probability in the retrieval results of each product in the product region is determined respectively, and a candidate item set for the product region is constructed. For example, select Figure 8 the item with the highest matching probability in the retrieval results of product 8501 in product region 850, such as sku4, the item with the highest matching probability in the retrieval results of product 8502, such as sku4, and the item with the highest matching probability in the retrieval results of product 8503, such as sku5 to construct the candidate item set for product region 850.

[0086] Then, the corresponding items in the candidate item set are voted according to the matching probabilities between the product and the corresponding items in the TopN recalled results of each product in the product region. Taking the matching results including 5 items in the retrieval results of products 8501, 8502, and 8503 as an example, that is, each product recalls 5 items, then the candidate item set includes 2 items (such as item sku4 and item sku5), and the item matching results of each product included in product region 850 are 15. Further, these 15 item matching results are used to vote on the 2 items included in the candidate item set. For example, the matching probabilities corresponding to item sku4 in these 15 item matching results are added up as the voting score of item sku4; the matching probabilities corresponding to item sku5 in these 15 item matching results are added up as the voting score of item sku5.

[0087] After that, the item with the highest voting score is selected as the item corresponding to the product region.

[0088] Further, in order to increase the candidate possibility of items with relatively high matching probabilities in the retrieval results, it is possible to further set gradually decreasing voting weights for the matching probabilities between the product and the corresponding items in the TopN recall results of each product. For example, the item with the highest matching probability with a certain product and its matching probability voting weight are set to 1, the item with the second-highest matching probability with this product and its matching probability voting weight are set to 0.5, and so on, gradually reducing the voting weights of the items and their matching probabilities at different sorting positions in the recall results.

[0089] Finally, aggregate the product regions of adjacent products corresponding to the same item.

[0090] Through retrieval voting, if the items corresponding to two product regions are the same, it indicates that the image regions corresponding to these two product regions describe the same product. Therefore, aggregation is required to obtain the final aggregation result. Suppose after retrieval voting, Figure 8 the items of product regions 810 and 820 in Figure 9 are the same, then product regions 810 and 820 are aggregated into one product region, and the final product aggregation result is as shown in

[0091] Further, according to the size and position information of the image regions obtained from the final aggregation result, it is possible to further determine the number of products of the corresponding item included in each aggregated image region.

[0092] The image-based product aggregation method disclosed in the embodiments of the present application, through left-merging and right-merging products based on the image features of adjacent products and further aggregating the merging results, does not need to compare with the absolute threshold of the feature distance, which helps to improve the accuracy of product aggregation. Finally, by voting on the items of each product in the product regions obtained by aggregation based on a preset standard item database, the items of each product region are determined. Then, based on the items of the product regions, adjacent product regions with the same item are further aggregated, which can further improve the accuracy of the product aggregation result.

[0093] Embodiment Three

[0094] In some other embodiments of the present application, when aggregating the image regions corresponding to the products according to the image features corresponding to the respective products to determine the product regions corresponding to the respective products, after determining the product regions corresponding to the respective products by aggregating the respective products through bilateral merging according to the distances between the image features of adjacent products in the image regions corresponding to the respective products, the method further includes: performing a segmentation process on the boundary products in the product regions corresponding to the respective products determined by aggregating the respective products through bilateral merging based on a relative threshold to adjust the product regions. Taking the product regions further aggregated after left merging and right merging in Embodiment 2 as an example, as Figure 8 For the product regions determined after aggregation in, for each product region, it is necessary to further determine whether there is a phenomenon of incorrect segmentation for the boundary products (i.e., the leftmost product and the rightmost product) in the product region.

[0095] In some embodiments of the present application, for each product region, the feature distance between every two products in the product region can be first calculated, and the minimum feature distance is selected as the comparison benchmark; then, the minimum feature distance between the boundary product in the product region and other products in the product region is compared with the minimum feature distance. If the comparison result exceeds a certain relative threshold, the boundary product is segmented to generate a new product region.

[0096] In some other embodiments of the present application, the step of splitting the boundary products in the product regions corresponding to the respective products determined by aggregating the respective products through bilateral merging based on a relative threshold and adjusting the product regions includes: performing the following operations on each of the product regions determined by aggregating the respective products through bilateral merging: determining the minimum feature distance between adjacent products in the product region, and determining the feature distances between the two boundary products in the product region and their respective adjacent products; determining the relative ratios of the feature distances between the two boundary products and their respective adjacent products to the minimum feature distance; splitting the boundary products that satisfy a preset condition for the magnitude relationship between the relative ratio and a preset relative ratio threshold to adjust the product region. For example, for the product region 850, first calculate the feature distances between product 8501 and product 8502, and between product 8502 and product 8503, respectively, and determine the minimum feature distance dist_interval_min; then, determine the relative ratio p1 of the feature distance dist_left between the boundary product 8501 and its adjacent product 8502 to the minimum feature distance dist_interval_min, and determine the relative ratio p2 of the feature distance dist_right between the boundary product 8503 and its adjacent product 8502 to the minimum feature distance dist_interval_min; if the relative ratio p1 is greater than the preset relative ratio threshold, split the boundary product 8501, that is, split the boundary product 8501 out of the product region 850, and generate a new product region from the boundary product 8501; if the relative ratio p2 is greater than the preset relative ratio threshold, split the boundary product 8503, that is, split the boundary product 8503 out of the product region 850, and generate a new product region from the boundary product 8503.

[0097] The image-based product aggregation method disclosed in this embodiment can improve the problem that when performing left and right merging on products and aggregating the product regions obtained by merging, only comparing the magnitude relationship of the feature distances of adjacent products cannot accurately aggregate the case where the differences between adjacent products gradually increase. By comprehensively considering the similarities of all products in the product region obtained by preliminary aggregation and further splitting and judging the boundary products, it helps to improve the accuracy of product aggregation. Moreover, the image-based product aggregation method disclosed in this embodiment only introduces a relative threshold within the product region during the over-segmentation judgment of the boundary products, and no threshold is introduced during the left and right merging based on the feature distance and the final product retrieval for the aggregation process. By introducing as few thresholds as possible, the robustness during the product aggregation process is improved.

[0098] Example 4

[0099] The product aggregation device based on images disclosed in the embodiments of the present application, as Figure 10 shown, the device includes:

[0100] A product image and feature determination module 1010, configured to determine the image regions and image features corresponding to each product included in the target image;

[0101] A product region determination module 1020, configured to aggregate the image regions corresponding to the products according to the image features corresponding to each product, and determine the product regions corresponding to each product;

[0102] A retrieval module 1030, configured to retrieve the product images of each product in a preset standard item database, and determine the item retrieval results of each product, where the product image is a part of the target image covered by the image region corresponding to the product;

[0103] A region aggregation module 1040, configured to aggregate the product regions corresponding to each product according to the item retrieval results, and determine the aggregation results of each product included in the target image.

[0104] Optionally, as Figure 11 shown, the product region determination module 1020 includes:

[0105] A first product region determination sub-module 1021, configured to aggregate each product included in the target image respectively through bilateral merging according to the distance between the image features of adjacent products in the image regions corresponding to each product, and determine the product regions corresponding to each product.

[0106] Optionally, the step of aggregating each product respectively through bilateral merging according to the distance between the image features of adjacent products in the image regions corresponding to each product, and determining the product regions corresponding to each product includes:

[0107] Left-merging each product respectively according to the distance between the image features of adjacent two products in each product to determine a number of first candidate product regions; and, right-merging each product respectively according to the distance between the image features of adjacent two products in each product to determine a number of second candidate product regions;

[0108] Aggregating the number of first candidate product regions and the number of second candidate product regions to determine the product regions corresponding to each product.

[0109] Optionally, as Figure 11As shown, after determining the product regions corresponding to the respective products by aggregating the respective products through bilateral merging based on the distances between the image features of adjacent products in the image regions corresponding to the respective products, the product region determination module 1020 further includes:

[0110] A boundary processing sub-module 1022, configured to perform segmentation processing on the boundary products in the product regions corresponding to the respective products determined by aggregating the respective products through bilateral merging based on a relative threshold, and adjust the product regions.

[0111] Optionally, the step of performing segmentation processing on the boundary products in the product regions corresponding to the respective products determined by aggregating the respective products through bilateral merging based on a relative threshold and adjusting the product regions includes:

[0112] Performing the following operations on each of the product regions determined by aggregating the respective products through bilateral merging:

[0113] Determining the minimum feature distance between adjacent products in the product region, and determining the feature distances between the two boundary products in the product region and their respective adjacent products;

[0114] Determining the relative ratios of the feature distances between the two boundary products and their respective adjacent products to the minimum feature distance;

[0115] Performing segmentation processing on the boundary products for which the magnitude relationship between the relative ratio and a preset relative ratio threshold satisfies a preset condition to adjust the product region.

[0116] Optionally, as Figure 12 shown, the product region determination module 1020 includes:

[0117] A second product region determination sub-module 1023, configured to aggregate the image regions corresponding to the products according to a preset feature distance threshold and the distances between the image features of adjacent products in the image regions corresponding to the respective products, and determine the product regions corresponding to the respective products.

[0118] Optionally, the item retrieval result includes: the item corresponding to the product, and the matching probability between the product and the corresponding item. The region aggregation module 1040 is further configured to:

[0119] For each of the product regions, perform item weighted voting based on the matching probability according to the item retrieval results of the respective products in the product region to determine the item corresponding to the product region;

[0120] Aggregate the product regions of adjacent and corresponding identical items, and determine the aggregation result of each product included in the target image.

[0121] The image-based product aggregation device disclosed in the embodiments of the present application is used to implement the steps of the image-based product aggregation method described in Embodiment 1 and Embodiment 2 of the present application. For the specific implementation manners of the modules of the device, refer to the corresponding steps, which will not be elaborated here.

[0122] The image-based product aggregation device disclosed in the embodiments of the present application determines the image regions and image features corresponding to each product in the target image; aggregates the image regions corresponding to the products according to the image features corresponding to the products to determine the product regions corresponding to the products; retrieves the product images of each product in a preset standard item database to determine the item retrieval results of each product, where the product image is a part of the target image covered by the image region corresponding to the product; aggregates the product regions corresponding to the products according to the item retrieval results to determine the aggregation result of each product included in the target image, and solves the problem of inaccurate product aggregation caused by comparing the feature distance and the threshold result in the prior art. Since the image-based product aggregation device disclosed in the embodiments of the present application sets a step of further correcting the aggregation result based on the image features through the retrieval result, the dependence on the absolute threshold is reduced, and the accuracy of the product aggregation result can be further improved.

[0123] The image-based product aggregation device disclosed in this embodiment can improve the left and right merging of products and the aggregation of the merged product regions by segmenting and judging the boundary products in the product regions formed by preliminary aggregation based on a relative threshold. When only comparing the magnitude relationship of the feature distances of adjacent products for product aggregation, it is impossible to accurately aggregate the situation where the differences between adjacent products gradually increase. By comprehensively considering the similarities of all products in the product regions obtained by preliminary aggregation and further segmenting and judging the boundary products, it helps to improve the accuracy of product aggregation. Moreover, in the image-based product aggregation method disclosed in this embodiment, only a relative threshold within the product region is introduced during the over-segmentation judgment of the boundary products, and no threshold is introduced during the left and right merging based on the feature distance and the final product retrieval for aggregation. By introducing as few thresholds as possible, the robustness during the product aggregation process is improved.

[0124] On the other hand, item voting is performed on the product areas obtained by aggregation based on the retrieval results of each product in the product areas obtained by aggregation determined according to a preset standard item database to determine the items of each product area. Then, based on the items of the product areas, adjacent product areas with the same items are further aggregated, which can further improve the accuracy of the product aggregation result.

[0125] Correspondingly, the present application also discloses an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the image-based product aggregation method described in Embodiments 1 to 3 of the present application. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc.

[0126] The present application also discloses a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps of the image-based product aggregation method described in Embodiments 1 to 3 of the present application.

[0127] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. The relevant parts can refer to the partial description of the method embodiments.

[0128] The above provides a detailed introduction to an image-based product aggregation method and device provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

[0129] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Claims

1. An image-based product aggregation method, characterized in that, Including: Determine the image regions and image features corresponding to each product included in the target image; according to the distances between the image features of adjacent products in the image regions corresponding to each product, aggregate each product respectively through bilateral merging to determine the product regions corresponding to each product, specifically including: when performing left merging, first traverse the products on this layer from the leftmost product on a layer of shelves from left to right, and sequentially calculate the image feature distance dist_left between this product and its adjacent product on the left and the image feature distance dist_right between this product and its adjacent product on the right. If the dist_left is greater than the dist_right, do not aggregate this product and generate a new product region. If the dist_left is less than the dist_right, aggregate this product with its adjacent product on the left, and then determine several first candidate product regions. When performing right merging, first traverse the products on this layer from the rightmost product on a layer of shelves from right to left, and sequentially calculate the image feature distance dist_left' between this product and its adjacent product on the left and the image feature distance dist_right' between this product and its adjacent product on the right. If the dist_left' is less than the dist_right', do not aggregate this product and generate a new product region. If the dist_left' is greater than the dist_right', aggregate this product with its adjacent product on the right, and then determine several second candidate product regions. Aggregate the several first candidate product regions and the several second candidate product regions to determine the product regions corresponding to each product. If a certain product is aggregated into the first candidate product region and the second candidate product region corresponding to different shelf positions, then aggregate the first candidate product region and the second candidate product region; retrieve the product images of each product in the preset standard item database to determine the item retrieval results of each product, where the product image is a part of the target image covered by the image region corresponding to the product; aggregate the product regions corresponding to each product according to the item retrieval results to determine the aggregation results of each product included in the target image.

2. The method according to claim 1, wherein The step of aggregating the image regions corresponding to the products according to the image features corresponding to each product to determine the product regions corresponding to each product further includes: performing segmentation processing on the boundary products in the product regions corresponding to each product determined by aggregating each product respectively through bilateral merging based on a threshold, and adjusting the product regions, where the boundary products are the leftmost product and the rightmost product in the product region.

3. The method according to claim 2, characterized in that The step of performing segmentation processing on the boundary products in the product regions corresponding to the respective products determined by aggregating the respective products through bilateral merging based on a threshold and adjusting the product regions includes: performing the following operations on each of the product regions determined by aggregating the respective products through bilateral merging: determining the minimum feature distance between adjacent products in the product region, and determining the feature distances between the two boundary products in the product region and their respective adjacent products; determining the ratios of the feature distances between the two boundary products and their respective adjacent products to the minimum feature distance; and performing segmentation processing on the boundary products that satisfy a preset condition regarding the magnitude relationship between the ratios and a preset ratio threshold to adjust the product regions.

4. The method according to any one of claims 1 to 3, characterized in that, The item retrieval result includes: the item corresponding to the product, and the matching probability between the product and the corresponding item. The step of aggregating the product regions corresponding to the respective products according to the item retrieval result and determining the aggregation result of each product included in the target image includes: for each of the product regions, performing item weighted voting based on the matching probability according to the item retrieval results of the respective products in the product region to determine the item corresponding to the product region; and aggregating the product regions that are adjacent and correspond to the same item to determine the aggregation result of each product included in the target image.

5. The method according to any one of claims 1 to 3, characterized in that, The step of determining the image regions and image features corresponding to the respective products in the target image includes: determining the image regions corresponding to the respective products included in the target image through a preset product detection model; and determining the image features of each of the image regions through a preset product feature extraction model.

6. An image-based product aggregation device, characterized in that, including: a product image and feature determination module, configured to determine the image regions and image features corresponding to the respective products included in the target image; A product area determination module, configured to aggregate each of the products respectively through bilateral merging according to the distances between the image features of adjacent products in the image areas corresponding to the respective products, and determine the product areas corresponding to the respective products. Specifically, it includes: when performing left merging, first traverse the products on this layer from left to right starting from the leftmost product on a layer of shelves, and sequentially calculate the image feature distance dist_left between this product and its adjacent product on the left and the image feature distance dist_right between this product and its adjacent product on the right. If the dist_left is greater than the dist_right, then do not aggregate this product and generate a new product area. If the dist_left is less than the dist_right, then aggregate this product with its adjacent product on the left, thereby determining a number of first candidate product areas. When performing right merging, first traverse the products on this layer from right to left starting from the rightmost product on a layer of shelves, and sequentially calculate the image feature distance dist_left' between this product on this layer and its adjacent product on the left and the image feature distance dist_right' between this product and its adjacent product on the right. If the dist_left' is less than the dist_right', then do not aggregate this product and generate a new product area. If the dist_left' is greater than the dist_right', aggregate this product with its adjacent product on the right, thereby determining a number of second candidate product areas. Aggregate the number of first candidate product areas and the number of second candidate product areas to determine the product areas corresponding to the respective products. If a certain product is aggregated into the first candidate product area and the second candidate product area corresponding to different shelf positions, then aggregate the first candidate product area and the second candidate product area; A retrieval module, configured to retrieve the product images of each of the products in a preset standard item database to determine the item retrieval results of each of the products, where the product image is a part of the target image covered by the image area corresponding to the product; A region aggregation module, configured to aggregate the product areas corresponding to the respective products according to the item retrieval results to determine the aggregation results of each of the products included in the target image.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image-based product aggregation method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the image-based product aggregation method according to any one of claims 1 to 5.

Images