Automobile part picture recognition method, device and equipment

Abstract

The invention discloses an automobile part picture recognition method, which comprises the following steps: acquiring an automobile part picture, and performing data cleaning processing on the automobile part picture to obtain a plurality of picture subsets; performing data annotation processing on the plurality of picture subsets to obtain picture training data; inputting the picture training data into a model constructed based on a ResNet network for training to obtain an accessory picture recognition model; performing global feature extraction on each picture in the picture training data through a preset global feature extraction model to obtain a global feature vector set; obtaining a to-be-recognized picture, and predicting the to-be-recognized picture through the accessory picture recognition model to obtain a recognition result; and when the identification result is judged to be a commodity picture, taking the identified accessory commodity picture as a return result, otherwise, by calculating the Euclidean distance between the global feature vector corresponding to the picture to be identified and each vector in the global feature vector set, determining an image retrieval result according to the Euclidean distance, and returning the image retrieval result to the user.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method, device and equipment for recognizing an image of an automobile part. Background Art

[0002] In the automotive parts management system (ERP), there are often scenarios where it is necessary to identify auto parts pictures. For example, when a customer needs to purchase a certain accessory product, or needs to quote a certain accessory product, the customer often uploads a picture of the auto part in a mobile app or mini program. If the auto parts management system can recognize the auto parts picture input by the user, then the automatic quotation of the accessory product can be realized. Secondly, in the warehouse management system (WMS), we can help manage inventory by identifying auto parts pictures. In addition, on the accessory product details page of the auto parts management system, we can use auto parts picture recognition to classify the pictures uploaded by users.

[0003] To realize the recognition of auto parts pictures, it is necessary to train the image recognition model, so it is also necessary to prepare the training data required for the training model. Although an auto parts management system usually contains a large number of parts pictures, there is no very accurate annotation data for these pictures, and there is a lot of interference data. For example, auto parts products include various sets of products, such as two-piece sets, three-piece sets, etc. A two-piece set is composed of two different parts, and a picture of a two-piece set is generally made by taking a picture of these two parts together. The lack of accurate annotation data and the existence of a large amount of interference data are a challenge to the recognition of auto parts pictures. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose an automobile parts image recognition method, device and equipment, aiming to solve the problems of low automobile parts image recognition accuracy and insufficient flexibility of the recognition mechanism in the prior art.

[0005] To achieve the above object, the present invention provides a method for identifying automobile parts images, the method comprising:

[0006] Obtaining automobile parts pictures, and performing data cleaning processing on the automobile parts pictures, including removing duplicate pictures, removing out-of-distribution pictures, and removing package products and non-product pictures, to obtain multiple picture subsets;

[0007] Performing data annotation processing including data set merging and data set annotation on the plurality of image subsets to obtain image training data including a commodity training subset and a non-commodity training subset;

[0008] Input the image training data into a model constructed based on a ResNet network for training to obtain an accessory image recognition model;

[0009] Performing global feature extraction on each image in the image training data using a preset global feature extraction model to obtain a set of global feature vectors including all images;

[0010] Acquire a picture to be identified, and predict the picture to be identified by using the accessory picture recognition model to obtain a recognition result;

[0011] When it is determined that the recognition result is a commodity image, the recognized accessory commodity image is returned as the result; otherwise, the Euclidean distance between the global feature vector corresponding to the image to be recognized and each vector in the global feature vector set is calculated, and the image retrieval result is determined according to the Euclidean distance and returned to the user.

[0012] Preferably, the auto parts pictures are subjected to data cleaning processing including deduplication of pictures, removal of out-of-distribution pictures, and removal of packaged goods and non-goods pictures, to obtain a plurality of picture subsets, including:

[0013] Deduplication of the automobile parts picture is performed using an image hash algorithm to obtain a first picture;

[0014] Extract features from the first picture to obtain feature vectors corresponding to each picture, and reduce the dimension of the feature vectors using a principal component analysis algorithm to obtain a second picture;

[0015] The second pictures are divided according to the same product to obtain a picture set corresponding to each product, and the feature vectors of the corresponding pictures in each picture set are clustered using a k-means clustering algorithm to obtain multiple first picture subsets;

[0016] Calculating a first distance from each picture in each of the first picture subsets to the center of the first picture subset, and calculating a variance of the first distance to obtain a subset variance;

[0017] Determine whether the subset variance is greater than a first threshold value, if so, move the first picture subset to a preset picture list, if not, determine whether the first distance exceeds a second threshold value, if so, use the corresponding picture as an out-of-distribution picture, and move the out-of-distribution picture from the first picture subset to the preset picture list to obtain a second picture subset;

[0018] Using a preset target detection model to detect each second image in each second image subset to obtain a detection result, and when it is determined that the product in the detection result is a package product, moving the second image to the preset image list to obtain a third image subset;

[0019] The non-product images in the third image subset are identified and moved to the preset image list to obtain the image subset.

[0020] Preferably, some pictures are obtained from the preset picture list, annotated and used as the non-commodity training subset.

[0021] Preferably, the image hash algorithm is a difference hash algorithm; the principal component analysis algorithm uses the decomposition.PCA class in the sklearn package.

[0022] Preferably, the data annotation processing including data set merging and data set annotation is performed on the plurality of image subsets to obtain image training data including commodity training subsets and non-commodity training subsets, including:

[0023] Predicting any two of the plurality of image subsets using a preset classification model to obtain a prediction result;

[0024] When it is determined that the probability that the labels corresponding to the two image subsets in the prediction result belong to the same label is greater than a threshold, the two image subsets are merged to obtain a merged image subset;

[0025] Each of the merged image subsets is labeled with different labels to obtain a plurality of the commodity category training subsets corresponding to different commodities.

[0026] Preferably, the classification model is obtained by fine-tuning the ResNet101V2 model in tensorflow.

[0027] Preferably, calculating the Euclidean distance between the global feature vector corresponding to the image to be identified and each vector in the global feature vector set, and determining the image retrieval result according to the Euclidean distance includes:

[0028] Obtaining multiple nearest neighbors of the global feature vector corresponding to the image to be identified by calculating the Euclidean distance between the global feature vector corresponding to the image to be identified and each vector in the global feature vector set;

[0029] Voting is performed according to the label corresponding to each nearest neighbor of the global feature vector of the image to be identified, the label corresponding to the nearest neighbor with the largest number of votes is taken as the target label, and the accessory product image corresponding to the target label is taken as the image retrieval result.

[0030] To achieve the above object, the present invention further provides an automobile parts image recognition device, the device comprising:

[0031] A first processing unit is used to obtain automobile parts pictures, and perform data cleaning processing on the automobile parts pictures, including removing duplicate pictures, removing out-of-distribution pictures, and removing package products and non-product pictures, to obtain multiple picture subsets;

[0032] A second processing unit is used to perform data labeling processing including data set merging and data set labeling on the plurality of image subsets to obtain image training data including a commodity training subset and a non-commodity training subset;

[0033] A model training unit, used for inputting the image training data into a model constructed based on a ResNet network for training to obtain an accessory image recognition model;

[0034] A feature extraction unit, configured to perform global feature extraction on each image in the image training data by using a preset global feature extraction model to obtain a set of global feature vectors including all images;

[0035] A recognition unit, used to obtain a picture to be recognized, and predict the picture to be recognized by using the accessory picture recognition model to obtain a recognition result;

[0036] The retrieval unit is used to determine that when the recognition result is a commodity image, the recognized accessory commodity image is returned as the result; otherwise, the Euclidean distance between the global feature vector corresponding to the image to be recognized and each vector in the global feature vector set is calculated, and the image retrieval result is determined according to the Euclidean distance and returned to the user.

[0037] In order to achieve the above objectives, the present invention also proposes an automobile parts image recognition device, including a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of an automobile parts image recognition method as described in the above embodiment.

[0038] In order to achieve the above object, the present invention further proposes a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the steps of a method for identifying automobile parts images as described in the above embodiment.

[0039] In order to achieve the above object, the present invention further proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a method for identifying automobile parts images as described in the above embodiment.

[0040] Beneficial effects:

[0041] The above solution optimizes the data processing process, achieves high intelligence and automation, reduces manual intervention, improves work efficiency and the recognition accuracy of auto parts pictures, and provides a flexible recognition mechanism. For pictures that cannot be accurately classified by the parts picture recognition model, another solution can be used for image retrieval, making the recognition of auto parts pictures more flexible and comprehensive. The solution is based on mature deep learning frameworks (such as tensorflow) and algorithms (such as ResNet, PCA, k-means, etc.), has good scalability and maintainability, and is easy to improve and optimize according to actual needs.

[0042] The above scheme effectively reduces the noise and interference in the training data through a series of data cleaning processes (including image deduplication, removal of out-of-distribution images, and removal of package products and non-product images), so that the subsequent image recognition model can be trained based on a purer and more relevant data set, thereby significantly improving the recognition accuracy and efficiency of auto parts images; the ResNet network is used to build an accessory image recognition model, and combined with data annotation processing (data set merging, data set annotation), it not only enhances the model's recognition ability for product images, but also introduces non-product training subsets to enable the model to better distinguish between product and non-product images, thereby improving the generalization and robustness of the model.

[0043] The above scheme, when the recognition result is a non-commodity image, implements feature-based image retrieval by calculating the Euclidean distance between the global feature vector of the image to be recognized and each vector in the global feature vector set. This mechanism not only provides users with an additional recognition path, but also helps users find the required accessories by retrieving similar images when accessories cannot be directly recognized, providing a flexible retrieval mechanism and enhancing the practicality and user experience of the system.

[0044] The above scheme optimizes the data cleaning and labeling process and improves processing efficiency and accuracy by utilizing image hashing algorithm for deduplication, principal component analysis algorithm for dimensionality reduction, and k-means clustering algorithm for clustering. At the same time, some images are obtained from the preset image list for labeling as non-commodity training subsets, further enriching the diversity of training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 The present invention is a flowchart of a method for identifying automobile parts images provided by an embodiment of the present invention.

[0047] Figure 2 A schematic diagram of a process for constructing image training data according to an embodiment of the present invention.

[0048] Figure 3 A schematic diagram of the structure of an automobile parts image recognition device provided by one embodiment of the present invention.

[0049] The realization of the purpose, functional features and advantages of the invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0051] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0052] The content of the present invention is described in detail below with reference to embodiments.

[0053] Reference Figure 1 FIG. 1 is a flow chart of a method for identifying automobile parts images provided in accordance with an embodiment of the present invention.

[0054] In this embodiment, the method includes:

[0055] S11, obtaining automobile parts pictures, performing data cleaning processing on the automobile parts pictures, including removing duplicate pictures, removing out-of-distribution pictures, and removing package products and non-product pictures, to obtain multiple picture subsets.

[0056] Furthermore, in step S11, the auto parts pictures are subjected to data cleaning processing including picture deduplication, removal of out-of-distribution pictures, and removal of packaged goods and non-goods pictures, to obtain a plurality of picture subsets, including:

[0057] S11-1, using an image hash algorithm to deduplicate the automobile parts picture to obtain a first picture;

[0058] S11-2, extracting features from the first picture to obtain a feature vector corresponding to each picture, and reducing the dimension of the feature vector using a principal component analysis algorithm to obtain a second picture;

[0059] S11-3, dividing the second pictures according to the same product to obtain a picture set corresponding to each product, and clustering the feature vectors of the corresponding pictures in each picture set using a k-means clustering algorithm to obtain multiple first picture subsets;

[0060] S11-4, calculating a first distance from each picture in each of the first picture subsets to the center of the first picture subset, and calculating a variance of the first distances to obtain a subset variance;

[0061] S11-5, determining whether the subset variance is greater than a first threshold, if so, moving the first picture subset to a preset picture list, if not, determining whether the first distance exceeds a second threshold, if so, treating the corresponding picture as an out-of-distribution picture, and moving the out-of-distribution picture from the first picture subset to the preset picture list to obtain a second picture subset;

[0062] S11-6, using a preset target detection model to detect each second image in each second image subset to obtain a detection result, and when it is determined that the product in the detection result is a package product, moving the second image to the preset image list to obtain a third image subset;

[0063] S11-7, identifying non-product images in the third image subset and moving them to the preset image list to obtain the image subset.

[0064] Furthermore, some pictures are obtained from the preset picture list, annotated and used as the non-commodity training subset.

[0065] In this embodiment, the method is applied to an automobile parts management system. An automobile parts management system will have a large number of users, and each user will have a series of automobile parts products. On the details page of each product, the user can upload multiple pictures related to the product. Therefore, an automobile parts management system will store a large number of automobile parts product pictures. A standard automobile parts product list is set in the automobile parts management system, and all users on this system need to manage their own products according to this product list. Then for each product in the product list, all pictures uploaded by users about this product can be collected, that is, each product corresponds to a picture list.

[0066] The product list is denoted as P. This list is fixed and its length is denoted as m. A product in this product list P is denoted as p i , that is, p i ∈P, i=1,…,m. Each product p i Each has a unique label (for example, the name of the product can be used as a label), denoted by l i At the same time, each product p i Each corresponds to a picture list, denoted as F i So, there are m picture lists F1, F2, ..., F m , corresponding to m products respectively. All pictures are denoted as F, that is, F = F1∪F2∪…∪F m In this way, we get a picture set F, and each picture in this picture set has a label. However, this picture set and the corresponding picture labels cannot be used as a training set because this picture set contains a lot of noise, so we need to clean the picture set F first. The data cleaning process includes:

[0067] Step 1.1: Image deduplication. The images uploaded by users contain many duplicate images. For example, for some common products, many users will use standard images on the Internet or platform, which will result in a large number of duplicate images. You can use the image hashing algorithm to deduplicate the images. Furthermore, the difference hashing algorithm can be used to deduplicate the images.

[0068] Step 1.2: Exclude out-of-distribution data. In order to exclude out-of-distribution (OOD) data, you need to extract the features of the image first, by using the deep neural network ResNet as the feature extractor of the image (such as using the pre-trained model ResNet50V2 of the tensorflow package). By removing the fully connected layer at the top of ResNet and replacing it with a global average pooling layer, where the global average pooling layer can use the GlobalAveragePooling2D layer in the tensorflow package. In this way, after feature extraction, each image obtains a vector, and the dimension of this vector is fixed, which is called the feature vector of the image. Then use the principal component analysis (PCA) algorithm to reduce the dimension of the feature vector of the image. The dimension after dimensionality reduction is a hyperparameter that needs to be set. The PCA algorithm can use the decomposition.PCA class in the sklearn package.

[0069] Although each product p i Each corresponds to a picture list (picture collection) F i , but not F i All the pictures in the same product are similar. For example, when a user uploads pictures for a product, he may upload pictures of the product taken from different angles. For example, the similarity between the front and side pictures of a car generator is very low. In addition, for the same product p i , the appearance of different brands may also be different. Therefore, for the same product p i Picture collection F i , it can also be subdivided into multiple subsets. Therefore, for each product p i Picture collection F i Perform clustering operations. Clustering algorithms can divide a data set into different classes or clusters. A parameter needs to be set for the clustering algorithm, namely the number of clusters. In other words, we hope that the clustering algorithm can divide a data set into several clusters, but we are not sure how many clusters each product image can be divided into. We can set the number of clusters based on the prior knowledge of each product image data. However, obtaining this prior knowledge requires a lot of manual operations. In order to minimize the manual operation, the number of clusters (denoted as c) is set to a larger value, such as c=30 or 50. This parameter c is the same for all m products.

[0070] For each image list F i , use the k-means clustering algorithm to cluster the feature vectors of the pictures. After the clustering algorithm, the picture list F i The images in will be assigned to c clusters, denoted as Gi1 , G i2 ,…,G ic , and call these c clusters the picture list F i c image subsets of G. ij The first subscript i in represents the index of the product i=1,…,m, and the second subscript j represents the index of the image subset j=1,…,c. Obviously, F i =G i1 ∪G i2 ∪…∪G ic . Because there are m products, each product has c image subsets, so there are a total of mc image subsets, such as Figure 2 shown.

[0071] For each image subset G ij , by calculating the distance from each image in this image subset to the center of the image subset G ij (cluster center) and calculate the variance of the distance. Because users can upload any picture, the picture set of each product has a certain amount of noise. The variance measures the degree of clustering of a distribution. When the variance is large, it means that the data points in this distribution are relatively scattered. So when a picture subset G ij When the variance of is large, it means that the images in this image subset vary greatly and there may be no common features among them. ij When the variance of G is greater than a threshold, the image subset will be removed. Specifically, by creating another image list Q and putting G ij Move the image in to Q, and set G ij For the image subset G whose variance meets the requirements ij , will calculate G ij The distance between each image in Q and the center of the image subset is calculated. If the distance between an image and the center of the image subset exceeds a threshold, the image is considered to be out of distribution and will be moved to Q. At the same time, the image will be moved from G ij Removed.

[0072] Step 1.3: Exclude packaged products. Object detection is used to exclude packaged products, so an object detection model needs to be trained first. Some pictures of packaged products need to be selected and labeled. In this embodiment, the object detection model is obtained based on YOLO training. ij For each image f in G, the target detection model will be used to predict image f. If two or more different products are detected in image f, image f will be removed from the image subset G. ijIt should be noted that the m products in the product list P are not package products, that is, each product p i They represent individual auto parts.

[0073] Step 1.4: Exclude non-product images. Common types of non-product images include product packaging boxes or labels. ERP system users sometimes add product packaging boxes or labels to the product details page, so these images need to be moved to the other image list Q. Of course, non-product images are not limited to packaging boxes or labels. Users can customize other non-product image categories.

[0074] This step can be done automatically, but first you need to manually select a number of non-product image subsets from the mc image subsets, and record the set of these non-product image subsets as X. And you need to select a number of non-product image subsets for each product p in the product list P. i Determine a subset of images, and record the set of these product image subsets as Y. At the same time, record all mc image subsets as a set W, that is, W = {G ij |i=1,2,…,m;j=1,2,…,c}, there is and And define the set Z = W-(X∪Y). Now for each image subset G in the set Z ij , if G ij The center of G is closer to the center of the image subset in set X than to the center of all image subsets in set Y. ij Move from set Z to set X, that is, update X = X∪{G ij} and Z=Z-{G ij}. Repeat the above operation until no new image subsets can be added to set X. The set X that is updated last is the set of image subsets that need to be excluded. For each image subset G in set X ij , will G ij Move all the pictures in to Q, and let

[0075] S12, performing data labeling processing including data set merging and data set labeling on the plurality of image subsets to obtain image training data including a commodity training subset and a non-commodity training subset.

[0076] Furthermore, in step S12, the data annotation processing including data set merging and data set annotation is performed on the plurality of image subsets to obtain image training data including a commodity training subset and a non-commodity training subset, including:

[0077] S12-1, predicting any two of the plurality of image subsets using a preset classification model to obtain a prediction result;

[0078] S12-2, when it is determined that the probability that the labels corresponding to the two image subsets in the prediction result belong to the same label is greater than a threshold, the two image subsets are merged to obtain a merged image subset;

[0079] S12-3, using different labels to mark each of the merged image subsets to obtain a plurality of the commodity category training subsets corresponding to different commodities.

[0080] In this embodiment, it is necessary to label the cleaned mc image subsets because there is noise in the image data. ij It is not certain that it is a product. i Therefore, each image subset needs to be labeled, that is, it needs to be labeled mc times. When m and c are relatively large, manual labeling mc times is also a relatively large task, so the mc image subsets will be automatically merged first. Specifically:

[0081] Step 2.1: Merge image subsets. First, divide the mc image subsets into c groups: the first group is (G 11 , G 21 ,…,G m1 ), the second group is (G 12 , G 22 ,…,G m2 ),…, the cth group is (G 1c , G 2c ,…,G mc ),like Figure 2 As shown. That is, each group has m image subsets, and each group contains only one image subset of a product. After grouping, the c image subsets of a product are assigned to different groups. By creating c image classification models ( Figure 2 The c classification models are trained with the data of the c image subsets. Because each image subset contains m categories, the output categories of each classification model are m. Image augmentation operations are performed before training. Image augmentation operations include horizontal flipping, vertical flipping, random scaling, arbitrary angle rotation, random translation, etc. Image augmentation operations can use functions such as RandomFlip, RandomZoom, and RandomRotation in tensorflow.keras.layers. The classification model can use the ResNet101V2 model in tensorflow, which is obtained by downloading the pre-trained model and then fine-tuning the pre-trained model.

[0082] After the c classification models are trained, they need to be used for prediction. Use each classification model to predict all images. Therefore, for each image subset G ij , there are c classification models that predict all the images in it. In general, consider the image subset G ij All the images in let the classification model k predict the result, where i = 1, ..., m; j = 1, ..., c; k = 1, ..., c. Only the case when j ≠ k needs to be considered. The training data used by the classification model k is the kth group of image subsets, i.e., G 1k , G 2k ,…,G mk Obviously, the labels of these image subsets are l1, l2, ..., l m . When the classification model k is used for the image subset G ij When predicting all the images in G ij It is predicted to be l i If the proportion of images in G reaches a preset threshold, such as 80%, then we say that the image subset G ik and G ij are similar, then we can ik and G ij Merging can make G ik =G ik ∪G ij , This merging can be done automatically by a computer.

[0083] Step 2.2: Image subset labeling. After all image subsets are merged, mc image subsets need to be labeled manually. Of course, empty image subsets will be ignored. After step 2.1 is completed, non-empty image subsets will account for a relatively small proportion. The specific manual operation is to assume that the non-empty image subset G ij All the pictures in the folder are placed in a folder. It is necessary to manually check the folder and confirm whether the pictures in the folder are consistent with the label li. If they are consistent, all the pictures in the folder are retained. If they are inconsistent, all the pictures in the folder are deleted.

[0084] After all image subsets are labeled, we can construct image training data for the accessory image recognition model. i , merge the image subset G after data cleaning and data annotation i1 , G i2 ,…,G ic To a set T i In the middle, it is called T i is a training subset, and the labels of all images in this set are li In this way, for m kinds of products, m training subsets T1, T2, ..., T m ,like Figure 2 shown.

[0085] Step 2.3: Construct the m+1th category. In order to distinguish the pictures of accessory products from other non-accessory products, the m+1th category needs to be constructed. The pictures in the m+1th category are all pictures of non-accessory products. You can select a part of the pictures from the other picture list Q as the m+1th category, but Q is also a very large set, so you can cluster Q again and divide Q into c′ clusters or subsets. In this way, you only need to mark c′ times, that is, for each subset of Q, you need to manually judge whether to mark the subset as the m+1th category.

[0086] The accessory image recognition model will contain m+1 categories, of which m categories are accessory products and 1 category is non-accessory products or other pictures. Through the second step, training data (including product training data and non-product data) is prepared for each category in the accessory image recognition model. Figure 2 shown.

[0087] S13, inputting the image training data into a model constructed based on a ResNet network for training to obtain an accessory image recognition model.

[0088] In this embodiment, the accessory image recognition model includes m+1 categories, and the accessory image recognition model adopts a standard model architecture. The backbone network of the model adopts ResNet, and the backbone network is followed by a fully connected layer. The model is trained using the training data prepared in the above steps to obtain the accessory image recognition model.

[0089] S14, performing global feature extraction on each picture in the picture training data through a preset global feature extraction model to obtain a global feature vector set including all pictures.

[0090] In this embodiment, the trained global feature extraction model is used to extract the Figure 2 The training subsets T1, T2, ..., T m ,T m+1 The global feature extraction model can use a pre-trained global feature extraction model, or it can be fine-tuned on the basis of a pre-trained model using training data to obtain a global feature extraction model suitable for the current business scenario. Specifically, for T i , i = 1, 2, ..., m, m + 1, each picture t ij (The first subscript indicates the i-th training subset, and the second subscript indicates the j-th image). ijInput the global feature extraction model, the model will output the global features of the image, the global feature is a vector of fixed length, denoted as v ij . In this way, the training subset T i The global feature vectors of each image in can form a set, denoted as V i , so there is v ij ∈V i By extracting global features from m+1 training subsets, we can obtain a set of m+1 global feature vectors, which are denoted as V1, V2, …, V m , V m+1 .

[0091] Since the m products in the product list P mentioned above are not set products, the accessory image recognition model trained in step 3.1 cannot recognize set products. To recognize set products by using image retrieval methods, it is necessary to prepare training data for each set product. The set of training images corresponding to a set product is also called a training subset. Assuming there are n sets of set products, there are n training subsets, denoted as S1, S2, ..., S n Similarly, we need to train the subsets S1, S2, ..., S offline. n Extract global features from each image in the training subset S i The set of global feature vectors generated by each picture in i=1, 2, ..., n is denoted as U i Therefore, for n training subsets, we get a set of n global feature vectors, which are denoted as U1, U2, ..., U n .

[0092] S15, obtaining a picture to be identified, and predicting the picture to be identified by using the accessory picture recognition model to obtain a recognition result.

[0093] S16, when it is determined that the recognition result is a commodity image, the recognized accessory commodity image is returned as the result; otherwise, by calculating the Euclidean distance between the global feature vector corresponding to the image to be recognized and each vector in the global feature vector set, the image retrieval result is determined according to the Euclidean distance and returned to the user, wherein the global feature vector corresponding to the image to be recognized is extracted by the global feature extraction model.

[0094] Furthermore, in step S16, calculating the Euclidean distance between the global feature vector corresponding to the image to be identified and each vector in the global feature vector set, and determining the image retrieval result according to the Euclidean distance includes:

[0095] S16-1, obtaining a plurality of nearest neighbors of the global feature vector corresponding to the image to be identified by calculating the Euclidean distance between the global feature vector corresponding to the image to be identified and each vector in the global feature vector set;

[0096] S16-2, voting is performed according to the label corresponding to each nearest neighbor of the global feature vector of the image to be identified, the label corresponding to the nearest neighbor with the largest number of votes is taken as the target label, and the accessory product image corresponding to the target label is taken as the image retrieval result.

[0097] In this embodiment, the image to be identified input by the user will first be classified. If the image cannot be classified, then image retrieval will be performed on it. The image to be identified input by the user is predicted by using the accessory image recognition model trained in step 3.1. If the image is classified as an accessory product image (i.e., the 1st, 2nd, ..., mth category), the result is directly returned to the user. If the image is classified as other images (i.e., the m+1th category) or the confidence of the image classification is lower than the preset threshold, then image retrieval is required for the image. In order to perform image retrieval on the image, the same global feature extraction model is used to extract the global features of the image and obtain the corresponding global feature vector, denoted as w. Then calculate w and the set V1, V2, ..., V m , V m+1 and the set U1, U2, ..., U n The Euclidean distance between each global feature vector in the (total m+n+1 sets) is used. Based on the Euclidean distance, we can get the top-k nearest neighbors of w. The label of w is determined by the nearest neighbor voting of w. Each nearest neighbor of vector w votes with its corresponding label, and the label with the most votes is used as the label of w.

[0098] In summary, this embodiment first uses the weak annotation information of auto parts product images (i.e., each image has a standard product category to which it belongs) to classify all images by product category, and then performs clustering operations on the images in each product. The annotation of images is converted into the annotation of image subsets, so that the total manual annotation workload is reduced by several orders of magnitude. This allows the annotation of massive image data to be completed efficiently.

[0099] This embodiment uses a method that combines image classification and image retrieval to complete the recognition of images. Image classification and image retrieval play a complementary role. The accessory image recognition model classifies images at the class level, while image retrieval classifies images at the instance level. Therefore, the image classification method is a relatively coarse-grained classification, while image retrieval is a fine-grained classification. The accessory image recognition model can be used to classify images with obvious category distinctions, and images that cannot be processed by the accessory image recognition model can be processed using a more detailed image retrieval. The accessory image recognition model can cover most of the images input by users, while image retrieval processes a small portion of the images.

[0100] Reference Figure 3 FIG. 1 is a schematic diagram of the structure of an automobile parts image recognition device provided by an embodiment of the present invention.

[0101] In this embodiment, the device 20 includes:

[0102] The first processing unit 21 is used to obtain automobile parts pictures, and perform data cleaning processing on the automobile parts pictures, including removing duplicate pictures, removing out-of-distribution pictures, and removing package products and non-product pictures, to obtain multiple picture subsets;

[0103] A second processing unit 22 is used to perform data labeling processing including data set merging and data set labeling on the plurality of image subsets to obtain image training data including a commodity training subset and a non-commodity training subset;

[0104] A model training unit 23 is used to input the image training data into a model constructed based on a ResNet network for training to obtain an accessory image recognition model;

[0105] A feature extraction unit 24 is used to perform global feature extraction on each picture in the picture training data through a preset global feature extraction model to obtain a global feature vector set including all pictures;

[0106] The recognition unit 25 is used to obtain a picture to be recognized, and predict the picture to be recognized by using the accessory picture recognition model to obtain a recognition result;

[0107] The retrieval unit 26 is used to determine that when the recognition result is a commodity image, the recognized accessory commodity image is returned as the result; otherwise, the Euclidean distance between the global feature vector corresponding to the image to be recognized and each vector in the global feature vector set is calculated, and the image retrieval result is determined according to the Euclidean distance and returned to the user.

[0108] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the above corresponding steps for details.

[0109] The embodiment of the present invention further provides an automobile parts picture recognition device, the device comprising the automobile parts picture recognition device as described above, wherein the automobile parts picture recognition device can be used Figure 3 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0110] The device includes: a mobile phone, a digital camera, a tablet computer, or other devices with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0111] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0112] The input unit can be used to receive input digital or character or image information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment includes not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0113] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel, and optionally, the display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface may cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event, and then the processor provides a corresponding visual output on the display panel according to the type of touch event.

[0114] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The computer readable storage medium may be a read-only memory, a disk or an optical disk.

[0115] The embodiment of the present invention also provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 A method for recognizing automobile parts images is shown.

[0116] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, equipment embodiment and storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0117] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0118] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for identifying automobile parts images, It is characterized in that The method comprises: Obtaining automobile parts pictures, and performing data cleaning processing on the automobile parts pictures, including removing duplicate pictures, removing out-of-distribution pictures, and removing package products and non-product pictures, to obtain multiple picture subsets; Performing data annotation processing including data set merging and data set annotation on the plurality of image subsets to obtain image training data including a commodity training subset and a non-commodity training subset; Input the image training data into a model constructed based on a ResNet network for training to obtain an accessory image recognition model; Performing global feature extraction on each image in the image training data using a preset global feature extraction model to obtain a set of global feature vectors including all images; Acquire a picture to be identified, and predict the picture to be identified by using the accessory picture recognition model to obtain a recognition result; When it is determined that the recognition result is a commodity image, the recognized accessory commodity image is returned as the result; otherwise, the Euclidean distance between the global feature vector corresponding to the image to be recognized and each vector in the global feature vector set is calculated, and the image retrieval result is determined according to the Euclidean distance and returned to the user.

2. The method for recognizing automobile parts images according to claim 1, It is characterized in that The data cleaning process of the auto parts pictures includes removing duplicate pictures, removing out-of-distribution pictures, and removing package products and non-product pictures to obtain multiple picture subsets, including: Deduplication of the automobile parts picture is performed using an image hash algorithm to obtain a first picture; Extract features from the first picture to obtain feature vectors corresponding to each picture, and reduce the dimension of the feature vectors using a principal component analysis algorithm to obtain a second picture; The second pictures are divided according to the same product to obtain a picture set corresponding to each product, and the feature vectors of the corresponding pictures in each picture set are clustered using a k-means clustering algorithm to obtain multiple first picture subsets; Calculating a first distance from each picture in each of the first picture subsets to the center of the first picture subset, and calculating a variance of the first distance to obtain a subset variance; Determine whether the subset variance is greater than a first threshold value, if so, move the first picture subset to a preset picture list, if not, determine whether the first distance exceeds a second threshold value, if so, use the corresponding picture as an out-of-distribution picture, and move the out-of-distribution picture from the first picture subset to the preset picture list to obtain a second picture subset; Using a preset target detection model to detect each second image in each second image subset to obtain a detection result, and when it is determined that the product in the detection result is a package product, moving the second image to the preset image list to obtain a third image subset; The non-product images in the third image subset are identified and moved to the preset image list to obtain the image subset.

3. The method for recognizing automobile parts images according to claim 2, It is characterized in that Some images are obtained from the preset image list and annotated as the non-commodity training subset.

4. The method for recognizing automobile parts images according to claim 2, It is characterized in that The image hash algorithm is a difference hash algorithm; the principal component analysis algorithm uses the decomposition.PCA class in the sklearn package.

5. The method for recognizing automobile parts images according to claim 1, It is characterized in that The step of performing data labeling processing including data set merging and data set labeling on the plurality of image subsets to obtain image training data including a commodity training subset and a non-commodity training subset includes: Predicting any two of the plurality of image subsets using a preset classification model to obtain a prediction result; When it is determined that the probability that the labels corresponding to the two image subsets in the prediction result belong to the same label is greater than a threshold, the two image subsets are merged to obtain a merged image subset; Each of the merged image subsets is labeled with different labels to obtain a plurality of the commodity category training subsets corresponding to different commodities.

6. The method for recognizing automobile parts images according to claim 5, It is characterized in that The classification model is obtained by fine-tuning the ResNet101V2 model in tensorflow.

7. The method for recognizing automobile parts images according to claim 1, It is characterized in that The step of calculating the Euclidean distance between the global feature vector corresponding to the image to be identified and each vector in the global feature vector set, and determining the image retrieval result according to the Euclidean distance includes: Obtaining multiple nearest neighbors of the global feature vector corresponding to the image to be identified by calculating the Euclidean distance between the global feature vector corresponding to the image to be identified and each vector in the global feature vector set; Voting is performed according to the label corresponding to each nearest neighbor of the global feature vector of the image to be identified, the label corresponding to the nearest neighbor with the largest number of votes is taken as the target label, and the accessory product image corresponding to the target label is taken as the image retrieval result.

8. An automobile parts image recognition device, It is characterized in that The device comprises: A first processing unit is used to obtain automobile parts pictures, and perform data cleaning processing on the automobile parts pictures, including removing duplicate pictures, removing out-of-distribution pictures, and removing package products and non-product pictures, to obtain multiple picture subsets; A second processing unit is used to perform data labeling processing including data set merging and data set labeling on the plurality of image subsets to obtain image training data including a commodity training subset and a non-commodity training subset; A model training unit, used for inputting the image training data into a model constructed based on a ResNet network for training to obtain an accessory image recognition model; A feature extraction unit, configured to perform global feature extraction on each image in the image training data by using a preset global feature extraction model to obtain a set of global feature vectors including all images; A recognition unit, used to obtain a picture to be recognized, and predict the picture to be recognized by using the accessory picture recognition model to obtain a recognition result; The retrieval unit is used to determine that when the recognition result is a commodity image, the recognized accessory commodity image is returned as the result; otherwise, the Euclidean distance between the global feature vector corresponding to the image to be recognized and each vector in the global feature vector set is calculated, and the image retrieval result is determined according to the Euclidean distance and returned to the user.

9. An automobile parts image recognition device, It is characterized in that The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of the automobile parts image recognition method as claimed in any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instruction, wherein the computer program / instruction is executed by the processor to implement the steps of the automobile parts image recognition method according to any one of claims 1 to 7.

Images