Commodity labeling method, device and storage medium

Abstract

The present application discloses a commodity labeling method, device and storage medium. The method includes: using a pre-trained target detection model to perform commodity detection on image information of a commodity to be labeled, obtaining commodity frame images of multiple commodity frames and initial category information corresponding to each commodity frame image; performing feature extraction on each commodity frame image, determining feature points and corresponding descriptors of each commodity frame image, matching feature points of two adjacent commodity frame images, and determining feature point matching distances of corresponding feature points; determining association relationships between commodity frames according to the detection results; determining a dictionary object for each commodity frame according to the detection results, feature point matching distances and association relationships; performing cluster analysis on multiple commodity frames based on the dictionary objects of all commodity frames using a preset clustering algorithm, and determining final category information for each commodity frame according to the clustering results.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a commodity labeling method, device and storage medium. Background Art

[0002] With the development of artificial intelligence and machine learning technologies, automated product labeling technologies have received widespread attention. These technologies are used in various fields, such as offline unmanned supermarkets and supermarket data analysis. Current labeling technologies mainly rely on manual labeling and some primary automated labeling tools. Manual labeling is the most traditional labeling method, relying on a large number of manual workers to label pictures one by one. Although the accuracy is high, the cost and time requirements are extremely high. Primary automated labeling tools use simple deep learning models, that is, they use some pre-trained neural networks for automatic labeling, such as YFaster R-CNN, etc. Although the efficiency has been improved, they perform poorly in labeling complex multimodal data. In addition, the combination of manual and automated labeling methods first uses an automatic algorithm for preliminary labeling, and then manually calibrates. This method still faces great challenges in terms of efficiency and cost.

[0003] Therefore, existing product labeling solutions have the following disadvantages: (1) High cost: The cost of manual labeling cannot be reduced, especially in large-scale data processing; (2) Low efficiency: Automatic labeling models are not efficient when processing complex multimodal data. Traditional methods rely on manual review and have low overall efficiency; (3) Time-consuming: The large-scale data labeling process is extremely time-consuming, affecting project progress; (4) Insufficient accuracy: Existing automatic labeling tools have much room for improvement in terms of technical complexity and labeling accuracy.

[0004] Existing product labeling methods have the problems of high cost, low efficiency, long time consumption and insufficient accuracy, and no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present disclosure provide a commodity labeling method, device and storage medium, so as to at least solve the technical problems of the commodity labeling methods existing in the prior art, such as high cost, low efficiency, long time consumption and insufficient accuracy.

[0006] According to one aspect of an embodiment of the present disclosure, a commodity labeling method is provided, comprising: using a pre-trained target detection model to perform commodity detection on image information of a commodity to be labeled, and obtaining a detection result, wherein the detection result includes commodity frame images of multiple commodity frames and initial category information corresponding to each of the commodity frame images; performing feature extraction on each commodity frame image, determining feature points and corresponding descriptors of each commodity frame image, and matching feature points of two adjacent commodity frame images to determine feature point matching distances of corresponding feature points; determining associations between commodity frames according to the detection result; determining a dictionary object for each commodity frame according to the detection result, the feature point matching distances and the associations; wherein the dictionary object is used to store information related to the commodity frame; based on the dictionary objects of all commodity frames, using a preset clustering algorithm, performing cluster analysis on the multiple commodity frames, and determining final category information of each commodity frame according to the clustering result.

[0007] Optionally, the feature extraction of each product frame image to determine the feature points and corresponding descriptors of each product frame image includes: converting each product frame image into a grayscale image to obtain a corresponding product frame grayscale image; and using a scale-invariant feature transformation algorithm to extract feature points and corresponding descriptors from each product frame grayscale image.

[0008] Optionally, determining the association relationship between the various product frames based on the detection result includes: calculating the center point coordinates of each product frame based on the detection result; calculating the center distance between any two product frames based on the center point coordinates; and determining the association relationship between the various product frames based on the center distance; wherein, when the center distance is less than or equal to a preset first threshold, there is an association relationship between the corresponding product frames, otherwise there is no association relationship.

[0009] Optionally, determining the dictionary object of each product frame according to the detection result, the feature point matching distance and the association relationship includes: dividing the multiple product frames into isolated product frames and non-isolated product frames according to the feature point matching distance and the association relationship; wherein, when the center distances of a product frame with all other product frames are greater than the first threshold and / or the feature point matching distances are greater than a preset second threshold, the product frame is an isolated product frame; otherwise, it is a non-isolated product frame; determining the dictionary objects of the isolated product frame and the non-isolated product frame according to the detection result, the feature point matching distance and the association relationship; wherein the key of the dictionary object of the isolated product frame is the index of the product frame, and the value is the coordinate information, height, width, initial category information, feature points and descriptors of the product frame; the key of the dictionary object of the non-isolated product frame is the index of the product frame, and the value is the coordinate information, height, width, initial category information of the product frame, the index of the target product frame associated with the product frame and the corresponding feature point matching distance.

[0010] Optionally, the dictionary object based on all the product frames uses a preset clustering algorithm to perform cluster analysis on the multiple product frames, and determines the final category information of each product frame according to the clustering result, including: constructing a feature matrix based on the dictionary object of all the product frames; wherein each row of the feature matrix represents a feature vector of a product frame, each column represents a feature, and the feature vector is obtained based on the dictionary object; using the OPTICS algorithm, performing cluster analysis on the feature matrix, and determining the final category information of each product frame according to the clustering result.

[0011] Optionally, the target detection model is trained in the following manner: sample images of goods are collected and the goods in the sample images are labeled; data enhancement is performed on the labeled sample images to obtain a labeled image data set; wherein the data enhancement operations include rotation, scaling, and flipping; and the YOLO model is trained using the image data set until the performance of the model meets the requirements to obtain the target detection model.

[0012] Optionally, determining the final category information of each product box according to the clustering result includes: assigning a label attribute to each cluster according to the clustering result; wherein the category corresponding to the label attribute is the final category information of the product box; and displaying the image information with the label attribute through a visualization tool.

[0013] According to another aspect of the embodiments of the present disclosure, a storage medium is further provided. The storage medium includes a stored program, wherein the above method is executed by a processor when the program is running.

[0014] According to another aspect of the embodiments of the present disclosure, a commodity labeling device is further provided, comprising: a detection module, configured to perform commodity detection on image information of a commodity to be labeled using a pre-trained target detection model, and obtain a detection result, wherein the detection result includes commodity frame images of multiple commodity frames and initial category information corresponding to each of the commodity frame images; a feature extraction module, configured to perform feature extraction on each commodity frame image, determine feature points and corresponding descriptors of each commodity frame image, match feature points of two adjacent commodity frame images, and determine feature point matching distances of corresponding feature points; determine association relationships between commodity frames according to the detection result; a dictionary object determination module, configured to determine a dictionary object for each commodity frame according to the detection result, the feature point matching distances and the association relationships; wherein the dictionary object is used to store information related to the commodity frame; and a clustering module, configured to perform cluster analysis on the multiple commodity frames using a preset clustering algorithm based on the dictionary objects of all commodity frames, and determine final category information of each commodity frame according to the clustering result.

[0015] According to another aspect of an embodiment of the present disclosure, a commodity labeling device is also provided, comprising: a processor; and a memory connected to the processor, and configured to provide the processor with instructions for processing the following processing steps: using a pre-trained target detection model to perform commodity detection on image information of a commodity to be labeled, and obtain a detection result, wherein the detection result includes commodity frame images of multiple commodity frames and initial category information corresponding to each of the commodity frame images; performing feature extraction on each commodity frame image, determining feature points and corresponding descriptors of each commodity frame image, and matching feature points of two adjacent commodity frame images to determine feature point matching distances of corresponding feature points; determining associations between the commodity frames according to the detection results; determining a dictionary object for each commodity frame according to the detection results, the feature point matching distances and the associations; wherein the dictionary object is used to store information related to the commodity frame; based on the dictionary objects of all commodity frames, using a preset clustering algorithm, performing cluster analysis on the multiple commodity frames, and determining final category information of each commodity frame according to the clustering results.

[0016] The technical solution of the present application first uses a pre-trained target detection model to perform product detection on the image information of the product to be marked, and obtains product frame images of multiple product frames and initial category information corresponding to each of the product frame images. Then, based on the output of the target detection model, the association relationship between each product frame and the feature point matching distance of adjacent product frames are determined. Secondly, according to the detection results, the feature point matching distance and the association relationship, the dictionary object of each product frame is determined to provide data support for subsequent clustering analysis. Finally, based on the dictionary objects of all product frames, a preset clustering algorithm is used to perform cluster analysis on the multiple product frames, and the final category information of each product frame is determined according to the clustering results. In this way, the accuracy of product labeling can be further optimized and verified based on the output of the target detection model, and efficient and accurate automatic identification and classification of products can be achieved, which not only reduces the need for manual labeling, but also improves processing speed and accuracy. Thereby solving the technical problems of high cost, low efficiency, long time consumption and insufficient accuracy of the product labeling method in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0018] Figure 1 is a flowchart of a commodity labeling method according to Embodiment 1 of the present disclosure;

[0019] Figure 2 is a schematic diagram of a commodity labeling device according to Embodiment 2 of the present disclosure; and

[0020] Figure 3 It is a schematic diagram of a commodity labeling device according to Embodiment 2 of the present disclosure. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0023] Example 1

[0024] According to this embodiment, a method embodiment of a commodity labeling method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0025] like Figure 1 As shown, according to this embodiment, a commodity labeling method is provided, including:

[0026] S102: Using a pre-trained target detection model, perform commodity detection on the image information of the commodity to be labeled to obtain a detection result, wherein the detection result includes commodity frame images of multiple commodity frames and initial category information corresponding to each of the commodity frame images.

[0027] Optionally, the target detection model is trained in the following manner: sample images of goods are collected and the goods in the sample images are labeled; data enhancement is performed on the labeled sample images to obtain a labeled image data set; wherein the data enhancement operations include rotation, scaling, and flipping; and the YOLO model is trained using the image data set until the performance of the model meets the requirements to obtain the target detection model.

[0028] Specifically, a labeled product dataset is needed to train the YOLO model. After training, a target detection model that can accurately detect products can be obtained. The dataset should contain images of products and corresponding category labels. For example, if you are processing images of supermarket products, each image in the dataset should be labeled with a specific product category, such as "apple", "milk", "bread", etc. Then, using YOLO's pre-trained model, this labeled dataset is integrated into the training process to ensure that the format of the dataset matches the input format expected by the model. Before training, data enhancement is performed on the dataset, including but not limited to rotation, scaling, and flipping. Among them, the probability of horizontal flipping is set to 0.36, that is, there is a 36% probability of horizontal flipping. The scale range of random cropping is set to (0.77, 1.0), that is, the size of the cropped image accounts for 77% to 100% of the original image size. The range of random scaling is set to (0.8, 1.2), that is, the size of the image will randomly change between 80% and 120% of the original size. By adjusting the above parameters, the model can better adapt to different product postures, sizes, lighting conditions, etc., improve the model's ability to detect products of different sizes, and reduce the risk of overfitting.

[0029] Then, the image information of the product to be labeled is obtained, and the trained object detection model is used to perform product detection on the image information of the product to be labeled, and the coordinate information of the product frame (also called product frame, detection frame), the corresponding initial category information and the confidence are output. The confidence is used to describe the probability that the model believes that the product frame contains a certain type of object. According to the product frame coordinate information output by the model, the product frame image of each product frame is cropped in the image information.

[0030] S104: extracting features from each product frame image, determining feature points and corresponding descriptors of each product frame image, matching feature points of two adjacent product frame images, and determining feature point matching distances of corresponding feature points; and determining association relationships between each product frame based on the detection results.

[0031] Optionally, the feature extraction of each product frame image to determine the feature points and corresponding descriptors of each product frame image includes: converting each product frame image into a grayscale image to obtain a corresponding product frame grayscale image; and using a scale-invariant feature transformation algorithm to extract feature points and corresponding descriptors from each product frame grayscale image.

[0032] Optionally, determining the association relationship between the various product frames based on the detection result includes: calculating the center point coordinates of each product frame based on the detection result; calculating the center distance between any two product frames based on the center point coordinates; and determining the association relationship between the various product frames based on the center distance; wherein, when the center distance is less than or equal to a preset first threshold, there is an association relationship between the corresponding product frames, otherwise there is no association relationship.

[0033] Specifically, although the model outputs the type information of the product frame, due to factors such as lighting, angle, occlusion, etc., the model's judgment may be inaccurate, and in complex scenes such as supermarket shelves, there may be occlusion, overlap or similar appearance between products. Relying solely on the category information output by the model may not accurately distinguish different products. Based on this, it is necessary to convert the product frame image detected by the model into a grayscale image to obtain MatGray (grayscale image matrix), and use the SIFT (scale-invariant feature transform) algorithm to extract feature points and descriptors from the grayscale image of the product frame. Among them, feature points are local areas in the image that are unique and stable, and descriptors are quantitative representations of the local areas around these feature points, which are used to describe the geometric and photometric properties of the feature points. The SIFT algorithm can detect and describe local features in images, which are invariant to changes in the scale, rotation and illumination of the image.

[0034] Then, the relationship between the product boxes is established. Each product box is represented as a four-tuple (x, y, w, h), where x and y represent the coordinates of the upper left corner of the box, and w and h represent the width and height of the box, respectively. All product boxes are collected in a list boxes in the form of boxes = [(x1, y1, w1, h1), (x2, y2, w2, h2), ...]. Assuming that each product box is a four-tuple (x, y, w, h), it is necessary to determine the association relationship between product box ① and product box ② in the two-dimensional plane. Specifically, first calculate the center points center1 and center2 of product box ① and product box ②, and calculate the center distance of the two product boxes distance = np.sqrt ((center1 [0] - center2 [0]) ** 2 + (center1 [1] - center2 [1]) ** 2). If the center distance of the two product boxes is less than the first threshold (avg_width + avg_height) / 2, the two product boxes are considered to be associated with each other. Among them, avg_width and avg_height refer to the average width and average height of all product boxes, which are used to set a reasonable distance threshold.

[0035] In addition, the BFMatcher (Brute Force Matcher) algorithm is used to match the feature points of each product frame and its adjacent frames. The BFMatcher algorithm is a brute force matching algorithm that traverses each descriptor in the query set and finds the closest training descriptor. During the matching process, the matching pairs whose distance is greater than the second threshold (for example, 0.78) are retained, and it is preliminarily assumed that the two product frames in these matching pairs belong to the same category.

[0036] Therefore, by extracting feature relationships under the same image, this method is considered more accurate than the traditional method of comparing features after sampling all images. This is because different images may have high noise, such as brightness differences, size ratio changes, etc., which may cause unoptimizable errors in the features extracted later. By extracting and comparing feature relationships within a single image, the influence of these external factors can be reduced, and the accuracy and reliability of feature matching can be improved.

[0037] S106: Determine a dictionary object for each product frame according to the detection result, the feature point matching distance and the association relationship; wherein the dictionary object is used to store information related to the product frame.

[0038] Optionally, determining the dictionary object of each product frame according to the detection result, the feature point matching distance and the association relationship includes: dividing the multiple product frames into isolated product frames and non-isolated product frames according to the feature point matching distance and the association relationship; wherein, when the center distances of a product frame with all other product frames are greater than the first threshold and / or the feature point matching distances are greater than a preset second threshold, the product frame is an isolated product frame; otherwise, it is a non-isolated product frame; determining the dictionary objects of the isolated product frame and the non-isolated product frame according to the detection result, the feature point matching distance and the association relationship; wherein the key of the dictionary object of the isolated product frame is the index of the product frame, and the value is the coordinate information, height, width, initial category information, feature points and descriptors of the product frame; the key of the dictionary object of the non-isolated product frame is the index of the product frame, and the value is the coordinate information, height, width, initial category information of the product frame, the index of the target product frame associated with the product frame and the corresponding feature point matching distance.

[0039] Specifically, after detection by the target detection model and feature extraction and matching, a dictionary object is constructed for each product frame to store information related to the product frame. These dictionary objects have different contents depending on the type of product frame (non-isolated product frame or isolated product frame). Among them, isolated product frames refer to those product frames that have no significant correlation with other product frames in the feature space, or cannot be effectively feature matched with other product frames due to some reasons (such as occlusion, insufficient lighting, angle problems, etc.). In other words, isolated product frames are product frames that are identified as independent in the feature matching process, and their correlation with other product frames is weak or non-existent.

[0040] For the dictionary object of a non-isolated product box, the key is the index (or unique identifier) ​​of the product box, and the value is a list containing the indexes of other product boxes associated with the product box and the corresponding feature point matching distances, etc. This list reflects the association and feature matching results between the product box and other product boxes. For the dictionary object of an isolated product box, the key is the index (or unique identifier) ​​of the product box, and the value is a list containing the feature points and descriptor set of the product box. This list reflects the uniqueness and independence of the product box in the feature space.

[0041] By constructing the above dictionary objects and list structures, we can systematically manage and analyze the product box information extracted from the image, whether it is those product boxes with clear adjacent relationships or those isolated product boxes. This organization method not only helps with subsequent feature analysis and classification tasks, but also provides flexible and powerful data support for product recognition in complex image scenes.

[0042] S108: Based on the dictionary objects of all the commodity frames, a preset clustering algorithm is used to perform clustering analysis on the multiple commodity frames, and final category information of each commodity frame is determined according to the clustering result.

[0043] Optionally, the dictionary object based on all the product frames uses a preset clustering algorithm to perform cluster analysis on the multiple product frames, and determines the final category information of each product frame according to the clustering result, including: constructing a feature matrix based on the dictionary object of all the product frames; wherein each row of the feature matrix represents a feature vector of a product frame, each column represents a feature, and the feature vector is obtained based on the dictionary object; using the OPTICS algorithm, performing cluster analysis on the feature matrix, and determining the final category information of each product frame according to the clustering result.

[0044] Optionally, determining the final category information of each product box according to the clustering result includes: assigning a label attribute to each cluster according to the clustering result; wherein the category corresponding to the label attribute is the final category information of the product box; and displaying the image information with the label attribute through a visualization tool.

[0045] Specifically, OPTICS (Ordering Points To Identify the Clustering Structure) is a density-based clustering algorithm that is particularly suitable for discovering clusters with different densities and processing noisy data. Before using the OPTICS algorithm for clustering analysis, the feature set of isolated product boxes is extracted. These features usually include the coordinates, scales, directions, and descriptors of the feature points. These feature vectors are used as input data for the OPTICS algorithm. Then, two key parameters of the OPTICS algorithm are set: the neighborhood radius E and the minimum number of domain points MinPts for a given point to become a core object in the E domain. The selection of these parameters usually needs to be adjusted according to the characteristics of the data and the clustering requirements. When performing clustering analysis, the OPTICS algorithm first calculates the distance between each pair of data points in the data set, usually using the Euclidean distance as a metric, and then calculates the core distance for each data point, that is, the minimum neighborhood radius required for the point to become a core object. Secondly, a neighborhood set for each data point is generated, which is a circular area centered on the data point with a radius equal to the core distance. In the cluster expansion phase, the algorithm divides the data points into core objects, boundary objects, and noise objects according to their reachable distance (i.e., the distance between the data points and the core objects). Finally, the clusters are expanded recursively and the boundary objects are assigned to the corresponding clusters until all data points have been visited. The OPTICS algorithm does not directly generate the resulting clusters, but generates an augmented cluster ordering for cluster analysis.

[0046] According to the cluster ranking generated by the OPTICS algorithm, different clusters can be identified. These clusters have similar characteristics in the feature space, reflecting the correlation between isolated product boxes. The size and shape of the clusters may vary, depending on the characteristics of the data and the settings of the clustering parameters. The product boxes within a cluster have high similarity in characteristics, while the product boxes between clusters have low similarity. By analyzing the clustering results, potential correlations and patterns between isolated product boxes can be discovered.

[0047] Furthermore, the information of all products (including non-isolated product boxes and isolated product boxes) is integrated into a feature matrix for more comprehensive clustering analysis. Specifically, a feature matrix is ​​constructed. Assuming that there are n product boxes, each product box has a feature vector feature_vector, then the feature matrix is ​​a two-dimensional array of shape (n, feature_dimension), where feature_dimension is the dimension of the feature vector. Among them, the feature vector feature_vector can include the category information, confidence, location information of the product box (such as center point coordinates, width and height) output by the target detection model, etc. For isolated product boxes, some clustering features calculated by the OPTICS algorithm (such as the distance to the nearest cluster, the identifier of the cluster to which it belongs, etc.) can also be included.

[0048] Then, the OPTICS algorithm is used to perform cluster analysis on the feature matrix. The algorithm first calculates the distance and density relationship between data points based on the eigenvalues ​​in the feature matrix. Then an ordered cluster sort is generated based on these distance and density relationships, and the clustering result is a label sequence, such as [0,0,1,1,-1,2,2,...]. Each label represents the cluster to which a product box belongs. Product boxes with the same label belong to the same cluster, and product boxes with different labels belong to different clusters. This means that the first and second product boxes belong to cluster 0, the third and fourth product boxes belong to cluster 1, the fifth product box is a noise point, the sixth and seventh product boxes belong to cluster 2, and so on.

[0049] Finally, according to the clustering categories, assign label attributes to the clustering categories and preview the results.

[0050] In addition, according to this embodiment, a storage medium is also provided, wherein the storage medium includes a stored program, wherein when the program is running, a processor executes any one of the above methods.

[0051] The present application first uses a pre-trained target detection model to perform commodity detection on the image information of the commodity to be marked, and obtains commodity frame images of multiple commodity frames and initial category information corresponding to each of the commodity frame images. Then, based on the output of the target detection model, the association relationship between each commodity frame and the feature point matching distance of adjacent commodity frames are determined. Secondly, according to the detection results, the feature point matching distance and the association relationship, the dictionary object of each commodity frame is determined to provide data support for subsequent clustering analysis. Finally, based on the dictionary objects of all commodity frames, a preset clustering algorithm is used to perform cluster analysis on the multiple commodity frames, and the final category information of each commodity frame is determined according to the clustering results. In this way, the accuracy of commodity labeling can be further optimized and verified on the basis of the output of the target detection model, and efficient and accurate automatic identification and classification of commodities can be achieved, which not only reduces the need for manual labeling, but also improves processing speed and accuracy. Thereby solving the technical problems of high cost, low efficiency, long time consumption and insufficient accuracy of the commodity labeling method in the prior art.

[0052] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0053] Example 2

[0054] Figure 2 FIG. 2 shows a schematic diagram of the structure of the commodity labeling device 200 according to this embodiment. Figure 2As shown, the commodity labeling device 200 includes: a detection module 210, which is used to use a pre-trained target detection model to perform commodity detection on the image information of the commodity to be labeled, and obtain a detection result, wherein the detection result includes commodity frame images of multiple commodity frames and initial category information corresponding to each of the commodity frame images; a feature extraction module 220, which is used to extract features from each commodity frame image, determine feature points and corresponding descriptors of each commodity frame image, match feature points of two adjacent commodity frame images, and determine feature point matching distances of corresponding feature points; determine the association relationship between each commodity frame according to the detection result; a dictionary object determination module 230, which is used to determine a dictionary object for each commodity frame according to the detection result, the feature point matching distance and the association relationship; wherein the dictionary object is used to store information related to the commodity frame; a clustering module 240, which is used to perform cluster analysis on the multiple commodity frames based on the dictionary objects of all commodity frames using a preset clustering algorithm, and determine the final category information of each commodity frame according to the clustering result.

[0055] Optionally, the feature extraction of each product frame image to determine the feature points and corresponding descriptors of each product frame image includes: converting each product frame image into a grayscale image to obtain a corresponding product frame grayscale image; and using a scale-invariant feature transformation algorithm to extract feature points and corresponding descriptors from each product frame grayscale image.

[0056] Optionally, determining the association relationship between the various product frames based on the detection result includes: calculating the center point coordinates of each product frame based on the detection result; calculating the center distance between any two product frames based on the center point coordinates; and determining the association relationship between the various product frames based on the center distance; wherein, when the center distance is less than or equal to a preset first threshold, there is an association relationship between the corresponding product frames, otherwise there is no association relationship.

[0057] Optionally, determining the dictionary object of each product frame according to the detection result, the feature point matching distance and the association relationship includes: dividing the multiple product frames into isolated product frames and non-isolated product frames according to the feature point matching distance and the association relationship; wherein, when the center distances of a product frame with all other product frames are greater than the first threshold and / or the feature point matching distances are greater than a preset second threshold, the product frame is an isolated product frame; otherwise, it is a non-isolated product frame; determining the dictionary objects of the isolated product frame and the non-isolated product frame according to the detection result, the feature point matching distance and the association relationship; wherein the key of the dictionary object of the isolated product frame is the index of the product frame, and the value is the coordinate information, height, width, initial category information, feature points and descriptors of the product frame; the key of the dictionary object of the non-isolated product frame is the index of the product frame, and the value is the coordinate information, height, width, initial category information of the product frame, the index of the target product frame associated with the product frame and the corresponding feature point matching distance.

[0058] Optionally, the dictionary object based on all the product frames uses a preset clustering algorithm to perform cluster analysis on the multiple product frames, and determines the final category information of each product frame according to the clustering result, including: constructing a feature matrix based on the dictionary object of all the product frames; wherein each row of the feature matrix represents a feature vector of a product frame, each column represents a feature, and the feature vector is obtained based on the dictionary object; using the OPTICS algorithm, performing cluster analysis on the feature matrix, and determining the final category information of each product frame according to the clustering result.

[0059] Optionally, the target detection model is trained in the following manner: sample images of goods are collected and the goods in the sample images are labeled; data enhancement is performed on the labeled sample images to obtain a labeled image data set; wherein the data enhancement operations include rotation, scaling, and flipping; and the YOLO model is trained using the image data set until the performance of the model meets the requirements to obtain the target detection model.

[0060] Optionally, determining the final category information of each product box according to the clustering result includes: assigning a label attribute to each cluster according to the clustering result; wherein the category corresponding to the label attribute is the final category information of the product box; and displaying the image information with the label attribute through a visualization tool.

[0061] Thus, according to this embodiment, firstly, a pre-trained target detection model is used to perform commodity detection on the image information of the commodity to be marked, and commodity frame images of multiple commodity frames and initial category information corresponding to each of the commodity frame images are obtained. Then, based on the output of the target detection model, the association relationship between each commodity frame and the feature point matching distance of adjacent commodity frames are determined. Secondly, according to the detection results, the feature point matching distance and the association relationship, the dictionary object of each commodity frame is determined to provide data support for subsequent clustering analysis. Finally, based on the dictionary objects of all commodity frames, a preset clustering algorithm is used to perform cluster analysis on the multiple commodity frames, and the final category information of each commodity frame is determined according to the clustering results. In this way, the accuracy of commodity labeling can be further optimized and verified based on the output of the target detection model, and efficient and accurate automatic identification and classification of commodities can be achieved, which not only reduces the need for manual labeling, but also improves the processing speed and accuracy. Thus, the technical problems of high cost, low efficiency, long time consumption and insufficient accuracy of the commodity labeling method in the prior art are solved.

[0062] Example 3

[0063] like Figure 3 As shown, this embodiment provides a commodity labeling device 300, including: a processor 310; and a memory 320, connected to the processor 310, and used to provide the processor 310 with instructions for processing the following processing steps: using a pre-trained target detection model, performing commodity detection on the image information of the commodity to be labeled, and obtaining a detection result, wherein the detection result includes commodity frame images of multiple commodity frames and initial category information corresponding to each of the commodity frame images; performing feature extraction on each commodity frame image, determining feature points and corresponding descriptors of each commodity frame image, and matching feature points of two adjacent commodity frame images to determine feature point matching distances of corresponding feature points; determining associations between the commodity frames according to the detection results; determining a dictionary object for each commodity frame according to the detection results, the feature point matching distances and the associations; wherein the dictionary object is used to store information related to the commodity frame; based on the dictionary objects of all commodity frames, using a preset clustering algorithm, performing cluster analysis on the multiple commodity frames, and determining final category information of each commodity frame according to the clustering results.

[0064] Optionally, the feature extraction of each product frame image to determine the feature points and corresponding descriptors of each product frame image includes: converting each product frame image into a grayscale image to obtain a corresponding product frame grayscale image; and using a scale-invariant feature transformation algorithm to extract feature points and corresponding descriptors from each product frame grayscale image.

[0065] Optionally, determining the association relationship between the various product frames based on the detection result includes: calculating the center point coordinates of each product frame based on the detection result; calculating the center distance between any two product frames based on the center point coordinates; and determining the association relationship between the various product frames based on the center distance; wherein, when the center distance is less than or equal to a preset first threshold, there is an association relationship between the corresponding product frames, otherwise there is no association relationship.

[0066] Optionally, determining the dictionary object of each product frame according to the detection result, the feature point matching distance and the association relationship includes: dividing the multiple product frames into isolated product frames and non-isolated product frames according to the feature point matching distance and the association relationship; wherein, when the center distances of a product frame with all other product frames are greater than the first threshold and / or the feature point matching distances are greater than a preset second threshold, the product frame is an isolated product frame; otherwise, it is a non-isolated product frame; determining the dictionary objects of the isolated product frame and the non-isolated product frame according to the detection result, the feature point matching distance and the association relationship; wherein the key of the dictionary object of the isolated product frame is the index of the product frame, and the value is the coordinate information, height, width, initial category information, feature points and descriptors of the product frame; the key of the dictionary object of the non-isolated product frame is the index of the product frame, and the value is the coordinate information, height, width, initial category information of the product frame, the index of the target product frame associated with the product frame and the corresponding feature point matching distance.

[0067] Optionally, the dictionary object based on all the product frames uses a preset clustering algorithm to perform cluster analysis on the multiple product frames, and determines the final category information of each product frame according to the clustering result, including: constructing a feature matrix based on the dictionary object of all the product frames; wherein each row of the feature matrix represents a feature vector of a product frame, each column represents a feature, and the feature vector is obtained based on the dictionary object; using the OPTICS algorithm, performing cluster analysis on the feature matrix, and determining the final category information of each product frame according to the clustering result.

[0068] Optionally, the target detection model is trained in the following manner: sample images of goods are collected and the goods in the sample images are labeled; data enhancement is performed on the labeled sample images to obtain a labeled image data set; wherein the data enhancement operations include rotation, scaling, and flipping; and the YOLO model is trained using the image data set until the performance of the model meets the requirements to obtain the target detection model.

[0069] Optionally, determining the final category information of each product box according to the clustering result includes: assigning a label attribute to each cluster according to the clustering result; wherein the category corresponding to the label attribute is the final category information of the product box; and displaying the image information with the label attribute through a visualization tool.

[0070] Thus, according to this embodiment, firstly, a pre-trained target detection model is used to perform commodity detection on the image information of the commodity to be marked, and commodity frame images of multiple commodity frames and initial category information corresponding to each of the commodity frame images are obtained. Then, based on the output of the target detection model, the association relationship between each commodity frame and the feature point matching distance of adjacent commodity frames are determined. Secondly, according to the detection results, the feature point matching distance and the association relationship, the dictionary object of each commodity frame is determined to provide data support for subsequent clustering analysis. Finally, based on the dictionary objects of all commodity frames, a preset clustering algorithm is used to perform cluster analysis on the multiple commodity frames, and the final category information of each commodity frame is determined according to the clustering results. In this way, the accuracy of commodity labeling can be further optimized and verified based on the output of the target detection model, and efficient and accurate automatic identification and classification of commodities can be achieved, which not only reduces the need for manual labeling, but also improves the processing speed and accuracy. Thus, the technical problems of high cost, low efficiency, long time consumption and insufficient accuracy of the commodity labeling method in the prior art are solved.

[0071] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0072] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0073] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A commodity labeling method, characterized in that: include: Using a pre-trained object detection model, perform commodity detection on the image information of the commodity to be labeled to obtain a detection result, wherein the detection result includes commodity frame images of a plurality of commodity frames and initial category information corresponding to each of the commodity frame images; Extract features from each product frame image, determine feature points and corresponding descriptors of each product frame image, match feature points of two adjacent product frame images, and determine feature point matching distances of corresponding feature points; determine association relationships between product frames based on the detection results; Determine a dictionary object for each product frame according to the detection result, the feature point matching distance and the association relationship; The dictionary object is used to store information related to the product frame; Based on the dictionary objects of all commodity frames, a preset clustering algorithm is used to perform clustering analysis on the multiple commodity frames, and final category information of each commodity frame is determined according to the clustering result.

2. The method according to claim 1, characterized in that The step of extracting features from each product frame image and determining feature points and corresponding descriptors of each product frame image includes: Convert each product frame image into a grayscale image to obtain the corresponding product frame grayscale image; Using the scale-invariant feature transformation algorithm, feature points and corresponding descriptors are extracted from the grayscale image of each product frame.

3. The method according to claim 1, characterized in that Determining the association relationship between the commodity boxes according to the detection result includes: Calculate the center point coordinates of each product frame according to the detection results; Calculate the center distance between any two product frames according to the center point coordinates; According to the center distance, the association relationship between each commodity frame is determined; wherein, when the center distance is less than or equal to a preset first threshold, there is an association relationship between the corresponding commodity frames, otherwise there is no association relationship.

4. The method according to claim 3, characterized in that The determining of the dictionary object of each product frame according to the detection result, the feature point matching distance and the association relationship includes: According to the feature point matching distance and the association relationship, the multiple product frames are divided into isolated product frames and non-isolated product frames; wherein, when the center distance of a product frame with all other product frames is greater than the first threshold and / or the feature point matching distance is greater than a preset second threshold, the product frame is an isolated product frame; otherwise, it is a non-isolated product frame; Determine dictionary objects of isolated product frames and non-isolated product frames according to the detection result, the feature point matching distance and the association relationship; wherein the key of the dictionary object of the isolated product frame is the index of the product frame, and the value is the coordinate information, height, width, initial category information, feature points and descriptors of the product frame; the key of the dictionary object of the non-isolated product frame is the index of the product frame, and the value is the coordinate information, height, width, initial category information of the product frame, the index of the target product frame that has an association relationship with the product frame and the corresponding feature point matching distance.

5. The method according to claim 1, characterized in that: The dictionary object based on all the commodity frames uses a preset clustering algorithm to perform clustering analysis on the multiple commodity frames, and determines the final category information of each commodity frame according to the clustering result, including: Based on the dictionary objects of all product frames, a feature matrix is ​​constructed; wherein each row of the feature matrix represents a feature vector of a product frame, each column represents a feature, and the feature vector is obtained based on the dictionary object; The OPTICS algorithm is used to perform cluster analysis on the feature matrix, and the final category information of each product box is determined according to the clustering result.

6. The method according to claim 1, characterized in that The object detection model is trained in the following way: Collect sample images of commodities and mark the commodities in the sample images; Performing data enhancement on the labeled sample images to obtain a labeled image data set; wherein the data enhancement operations include rotation, scaling, and flipping; The YOLO model is trained using the image dataset until the performance of the model meets the requirements, thereby obtaining the target detection model.

7. The method according to claim 1, characterized in that Determining the final category information of each product box according to the clustering result includes: According to the clustering result, a label attribute is assigned to each cluster; wherein the category corresponding to the label attribute is the final category information of the product box; Display image information with label attributes through visualization tools.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the processor executes the method according to any one of claims 1 to 7.

9. A commodity labeling device, characterized in that: include: A detection module, configured to perform commodity detection on the image information of the commodity to be labeled using a pre-trained target detection model, and obtain a detection result, wherein the detection result includes commodity frame images of a plurality of commodity frames and initial category information corresponding to each of the commodity frame images; A feature extraction module is used to extract features from each product frame image, determine feature points and corresponding descriptors of each product frame image, match feature points of two adjacent product frame images, and determine feature point matching distances of corresponding feature points; and determine associations between product frames based on the detection results; A dictionary object determination module, used to determine the dictionary object of each commodity frame according to the detection result, the feature point matching distance and the association relationship; The dictionary object is used to store information related to the product frame; The clustering module is used to perform clustering analysis on the multiple commodity frames based on the dictionary objects of all commodity frames by using a preset clustering algorithm, and determine the final category information of each commodity frame according to the clustering result.

10. A commodity labeling device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Using a pre-trained object detection model, perform commodity detection on the image information of the commodity to be labeled to obtain a detection result, wherein the detection result includes commodity frame images of a plurality of commodity frames and initial category information corresponding to each of the commodity frame images; Extract features from each product frame image, determine feature points and corresponding descriptors of each product frame image, match feature points of two adjacent product frame images, and determine feature point matching distances of corresponding feature points; determine association relationships between product frames based on the detection results; Determine a dictionary object for each product frame according to the detection result, the feature point matching distance and the association relationship; The dictionary object is used to store information related to the product frame; Based on the dictionary objects of all commodity frames, a preset clustering algorithm is used to perform clustering analysis on the multiple commodity frames, and final category information of each commodity frame is determined according to the clustering result.

Images