Image database construction, retrieval and training methods and devices, medium and product

Abstract

The invention discloses an image feature database construction method, an image retrieval method, a training method, an electronic device, a computer readable storage medium and a computer program product. The construction method comprises the steps of extracting a first image feature of a preset image based on a pre-trained first embedding model; and constructing an image feature database according to the first image feature. According to the construction method, the first feature of the reference image is extracted through the first embedded model, the first feature is stored in the image feature database, feature extraction and storage of the preset image can be automatically achieved, construction of the image feature database is completed, and therefore the image feature database can be constructed in a feature comparison mode based on the image feature database. And automatic, efficient and accurate image retrieval is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image retrieval, and more particularly, to a method for constructing an image feature database, an image retrieval method, a training method, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] Image matching retrieval is an important research direction in the field of computer vision, aiming to quickly find the most similar image to the query image from a large-scale image library. When retrieving images, an image database that can support fast retrieval needs to be constructed. In related technologies, how to construct a database that can achieve efficient and accurate image matching retrieval is a problem to be solved. SUMMARY

[0003] The present application provides a method for constructing an image feature database, an image retrieval method, a training method, an electronic device, a computer readable storage medium, and a computer program product.

[0004] The present application provides a method for constructing an image feature database, the method comprising:

[0005] Based on the first embedding model pre-trained, the first image feature of the preset image is extracted;

[0006] According to the first image feature, the image feature database is constructed.

[0007] Thus, in the construction method, image retrieval method, training method, electronic device, computer readable storage medium and computer program product of the present application, the first feature of the reference image is extracted by the first embedding model, and the first feature is stored in the image feature database. The feature extraction and storage of the preset image can be automatically realized, and the construction of the image feature database is completed, so that based on the image feature database, the automatic and efficient and accurate image retrieval can be realized by the feature comparison method.

[0008] In some embodiments, the construction method further comprises:

[0009] Based on the preset indexing method, the preset image feature database is constructed;

[0010] According to the first image feature, the image feature database is constructed, comprising:

[0011] According to the first image feature and the preset image feature database, the image feature database is constructed.

[0012] In some embodiments, the construction method further comprises:

[0013] a database environment for constructing the image feature database;

[0014] In the database environment, the preset image feature database is initialized according to a preset storage hyperparameter.

[0015] In some embodiments, the construction method further comprises:

[0016] establishing a feature link between the preset image and the first image feature, the feature link being used for determining the corresponding preset image according to the first image feature.

[0017] The embodiments of the present application provide an image retrieval method based on the image feature database constructed by the construction method of any of the above embodiments, the image retrieval method comprising:

[0018] extracting a first feature in a reference image based on the first embedding model;

[0019] retrieving a target image associated with the reference image according to the first feature based on the image feature database.

[0020] In some embodiments, the retrieving a target image associated with the reference image according to the first feature based on the image feature database comprises:

[0021] determining a target image feature corresponding to the first feature in the image feature database according to the first feature;

[0022] determining the target image according to the target image feature.

[0023] In some embodiments, the determining a target image feature corresponding to the first feature in the image feature database according to the first feature comprises:

[0024] determining a target image feature corresponding to the first feature in the image feature database according to the first feature based on a preset indexing method.

[0025] In some embodiments, the determining a target image feature corresponding to the first feature in the image feature database according to the first feature based on a preset indexing method comprises:

[0026] determining the target image feature according to a similarity between an image feature in the image feature database and the first feature based on a preset indexing method.

[0027] In some embodiments, the image retrieval method further comprises:

[0028] Based on the first embedding model, a category feature corresponding to the reference image is determined, and the category feature is used to determine the reliability of the first feature.

[0029] The embodiments of the present application provide a training method for training the first embedding model according to any one of the above embodiments, and the training method comprises the following steps:

[0030] The original image is preprocessed to obtain a training image set.

[0031] Based on a preset optimization method, the preset model is trained according to the training image set to obtain the first embedding model.

[0032] In some embodiments, the preprocessing of the original image to obtain the training image set comprises:

[0033] The original image is preprocessed to obtain a processed image.

[0034] The processed image is classified to obtain the training image set.

[0035] In some embodiments, the preprocessing of the original image comprises:

[0036] The original image is compressed to a preset size to obtain a compressed image.

[0037] The compressed image is normalized and / or data enhanced to complete the preprocessing of the original image.

[0038] In some embodiments, the training of the preset model according to the training image set comprises:

[0039] The preset model is trained according to the training image set and a focal loss function.

[0040] In some embodiments, the original image satisfies at least one of a preset proportion condition, a preset angle condition, a preset light condition and a preset quality condition.

[0041] In some embodiments, the preset proportion condition comprises that a proportion of a region occupied by a target object in the original image is within a preset proportion range; and / or,

[0042] The preset quality condition comprises that a resolution of the original image is a preset resolution.

[0043] The embodiments of the present application provide an electronic device, which comprises one or more processors and a memory, and the memory stores a computer program. When the computer program is executed by the processor, the steps of the method according to any one of the above embodiments are implemented.

[0044] The embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the program realizes the steps of the method of any of the above embodiments when executed by a processor.

[0045] The embodiments of the present application provide a computer program product, which comprises a computer program, and the program realizes the steps of any of the above embodiments when executed by a processor.

[0046] Additional aspects and advantages of the present application will be made apparent from the following description of the embodiments of the present application, which will be given by way of example only. BRIEF DESCRIPTION OF DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description of the embodiments of the present application, which will be given by way of example only, with reference to the following drawings, in which:

[0048] Figure 1 is a flowchart of a construction method of some embodiments of the present application;

[0049] Figure 2 is a flowchart of a construction method of some embodiments of the present application;

[0050] Figure 3 is a schematic diagram of an HSNW indexing method of some embodiments of the present application;

[0051] Figure 4 is a flowchart of a construction method of some embodiments of the present application;

[0052] Figure 5 is a flowchart of a construction method of some embodiments of the present application;

[0053] Figure 6 is a flowchart of an image retrieval method of some embodiments of the present application;

[0054] Figure 7 is a flowchart of an image retrieval method of some embodiments of the present application;

[0055] Figure 8 is a flowchart of an image retrieval method of some embodiments of the present application;

[0056] Figure 9 is a flowchart of an image retrieval method of some embodiments of the present application;

[0057] Figure 10 is a flowchart of an image retrieval method of some embodiments of the present application;

[0058] Figure 11is a flowchart of a training method of some embodiments of the present application;

[0059] Figure 12 is a flowchart of a training method of some embodiments of the present application;

[0060] Figure 13 is a flowchart of a training method of some embodiments of the present application;

[0061] Figure 14 is a flowchart of a training method of some embodiments of the present application;

[0062] Figure 15 is a schematic diagram of a preset angle in a spherical coordinate system according to some embodiments of the present application. DETAILED DESCRIPTION

[0063] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which the same or similar components have the same or similar designations and functions throughout, and embodiments are described by way of example in the accompanying drawings in which the same or similar components have the same or similar designations and functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are for the purpose of explanation only, and are not to be understood as limiting the present application.

[0064] Image matching retrieval is an important research direction in the field of computer vision, aiming to quickly find the most similar image to the query image from a large-scale image library. When retrieving images, an image database capable of supporting fast retrieval needs to be constructed. In the related art, how to construct a database capable of realizing efficient and accurate image matching retrieval is a problem to be solved.

[0065] Referring to Figure 1 , based on the above-mentioned problems to be solved, the embodiments of the present application provide a method for constructing an image feature database, the method comprising:

[0066] 011: extracting first image features of a preset image based on a first embedding model whose pre-training is completed;

[0067] 012: constructing an image feature database according to the first image features.

[0068] The embodiments of the present application provide an electronic device, which comprises one or more processors and a memory, and the memory stores a computer program. When the computer program is executed by the processor, the processor can realize the steps of the method according to any embodiment of the present application. That is, the processor can be used to extract first image features of a preset image based on a first embedding model whose pre-training is completed; and construct an image feature database according to the first image features.

[0069] The embodiment of the application provides a construction device, which comprises a first extraction module and a first construction module. The first extraction module is configured to extract first image features of preset images based on a first embedding model. The first construction module is configured to construct an image feature database according to the first image features.

[0070] Specifically, the first embedding model comprises a ViT model, a SE-Net model, a CBAM model and the like. In the embodiment of the application, when the image classification model is used as the first embedding model, the classification layer at the end of the image classification model is removed, and the first image features of the preset images are extracted only by using the image classification model.

[0071] The image feature database comprises a Faiss vector database, a MiLvus vector database, a Pinecone vector database and the like.

[0072] The preset images can be various types of images containing target objects collected in advance. The target objects can be various things that users are interested in. For example, the target objects can be vehicle parts and accessories, furniture, plants, jewelry and the like.

[0073] In the embodiment, the first embedding model is a ViT-B model, the image feature database is a Faiss vector database, and the target object is a vehicle part. That is, the ViT-B embedding network model with the removed classification layer is used to extract features of image data of parts and accessories of various vehicle brands, to obtain the first image features, and the Faiss vector database is used to embed the first image features, so as to complete construction of the image feature database. The vector database Faiss is a high-efficiency similarity search and clustering library developed by the Facebook AI Research Institute. It can quickly process large-scale data and support similarity search in a high-dimensional space.

[0074] The construction method of the embodiment of the application extracts first image features of vehicle parts and region images of various brands based on a ViT image embedding model, and uses a Faiss vector database storage technology to store feature vectors of the images, so that an automatic data storage process is constructed, and the cumbersome steps of construction of a traditional database are greatly reduced.

[0075] In this way, the first features of the reference images are extracted by the first embedding model, and the first features are stored in the image feature database, so that the feature extraction and storage of the preset images can be automatically realized, the construction of the image feature database is completed, and therefore, based on the image feature database, image retrieval can be realized in an automatic, efficient and accurate manner through feature comparison.

[0076] Please refer to Figure 2In some embodiments, the constructing method further comprises:

[0077] 013: constructing a preset image feature database based on a preset indexing method;

[0078] Step 012: constructing an image feature database according to the first image features, comprising:

[0079] 0121: constructing the image feature database according to the first image features and the preset image feature database.

[0080] In some embodiments, the processor can be configured to construct a preset image feature database based on a preset indexing method, and construct an image feature database according to the first image features and the preset image feature database.

[0081] In some embodiments, the constructing device comprises a second constructing module, and the first constructing module comprises a first constructing submodule. The second constructing module is configured to construct a preset image feature database based on a preset indexing method, and the first constructing submodule is configured to construct an image feature database according to the first image features and the preset image feature database.

[0082] Specifically, the preset indexing method comprises HNSW (Hierarchical Navigable Small World), NSG (Navigating Spreading-out Graph), KGraph, K-Means Tree, etc. In this embodiment, the preset indexing method is taken as HNSW for illustration. The preset image feature database is constructed based on the preset indexing method, so that the preset image feature database can support indexing using the preset indexing method.

[0083] The HNSW indexing method is a retrieval method based on a multi-layer graph, and retrieval is performed from the upper layer to the lower layer, and the number of nodes and the number of links of the upper layer are more sparse than those of the lower layer. The structure is as shown in Figure 3 The principle is similar to that of a multi-level skip list query. Initially, similar feature vectors are found based on sparse links, and then search is performed layer by layer downward, which can greatly improve the retrieval speed and efficiency.

[0084] In one embodiment, the first image features of the extracted automobile parts and accessories are added to the Faiss vector database constructed based on the HNSW index, and the Faiss vector database construction of the automobile parts and accessories is completed. Further, the HNSW approximate nearest neighbor vector retrieval method of the graph can be used for image matching retrieval of the Faiss vector database.

[0085] Thus, the image feature database is constructed based on the preset indexing method, so that the image feature database can support a more efficient preset indexing method, thereby enabling retrieval based on the image feature database based on the preset indexing method, and further accelerating the retrieval speed.

[0086] Referring to Figure 4 In some embodiments, the construction method further comprises:

[0087] 014: constructing a database environment of the image feature database;

[0088] 015: initializing the preset image feature database according to preset storage hyperparameters in the database environment.

[0089] In some embodiments, the processor can be configured to construct a database environment of the image feature database; and initialize the preset image feature database according to preset storage hyperparameters in the database environment.

[0090] In some embodiments, the construction device further comprises a third construction module and an initialization module. The third construction module can be configured to construct a database environment of the image feature database; and the initialization module can be configured to initialize the preset image feature database according to preset storage hyperparameters in the database environment.

[0091] Specifically, the database environment of the image feature database is constructed to facilitate the construction of the database at a later stage. For example, the Faiss-gpu or Faiss-cpu dependent library is downloaded in a specific environment to construct the environment of the Faiss vector database.

[0092] After the database environment is constructed, the preset storage hyperparameters are set to initialize the preset image feature database. The preset storage hyperparameters are pre-set parameters mainly used to affect the performance and resource utilization of the image feature database. The preset image feature database is initialized using the preset storage hyperparameters, which can limit the indexing method of the preset image database according to the demand.

[0093] In one embodiment, a preset Faiss vector database based on HNSW indexing is initialized in the Faiss vector library environment. The corresponding storage hyperparameters include the number of storage vectors sum, the number of neighbors added to each vector m, the number of vector entity nodes of each layer vector_layer_sum, and the number of vector nodes used by each layer in the process of constructing the vector database.

[0094] Thus, by constructing the database environment of the characteristic feature database, the running environment of the image feature database can be obtained, and the image feature database is initialized according to the preset storage hyperparameter to obtain the preset image feature database that can satisfy the preset indexing mode, so as to facilitate subsequent construction of the image feature database.

[0095] Referring to Figure 5 In some embodiments, the construction method further comprises:

[0096] 016: establishing a feature link between the preset image and the first image feature, and the feature link can be used to determine the corresponding preset image according to the first image feature.

[0097] In some embodiments, the processor can be used to establish a feature link between the preset image and the first image feature.

[0098] In some embodiments, the construction device further comprises an establishing module. The establishing module can be used to establish a feature link between the preset image and the first image feature.

[0099] Specifically, the first image feature stored in the image feature database and the first image feature corresponding thereto are established a feature link, so that the corresponding preset image can be determined according to the first image feature. It should be noted that the first image feature is extracted from the preset image, that is, one first image feature must exist a unique preset image corresponding thereto, and one preset image may correspond to multiple first image features. Therefore, according to the first image feature, the preset image can be determined.

[0100] In one embodiment, the preset image is an original acquisition image of various car brands. An image folder for image retrieval is constructed, and the original acquisition image is stored in the image folder, which can be used to determine the corresponding original acquisition image according to the first image feature in the image feature database obtained by image retrieval. The original acquisition image is processed to obtain a picture size [224, 224] that can be input to the ViT-B model for feature extraction, and is input to the trained ViT-B model for image 2048-dimensional feature extraction. The obtained first image feature is stored in the preset image feature database, and a link with the image folder is established, and the construction of the image feature database is completed.

[0101] Thus, by establishing a feature link between the preset image and the first image feature, in the case of retrieving the first image feature corresponding to the feature of the reference image based on the reference image, the corresponding preset image can be determined according to the feature link of the first image feature, so as to complete the retrieval of the reference image.

[0102] Referring to Figure 6The embodiment of the present application provides a kind of image retrieval method, image retrieval method is based on the image feature database constructed as any one of the above embodiment, image retrieval method includes:

[0103] 021: based on the first embedding model, the first feature in the reference image is extracted;

[0104] 022: based on the image feature database, according to the first feature, the target image associated with the reference image is retrieved.

[0105] The embodiment of the present application provides an electronic device, and the electronic device includes one or more memories and processors. The memory stores a computer program that can be executed by the processor. The processor can be used to extract the first feature in the reference image based on the first embedding model;Based on the image feature database, according to the first feature, the target image associated with the reference image is retrieved.

[0106] The embodiment of the present application provides an image retrieval device, and the image retrieval device includes a second extraction module and a retrieval module. Wherein, the second extraction module can be used to extract the first feature in the reference image based on the first embedding model;The retrieval module can be used to retrieve the target image associated with the reference image based on the image feature database according to the first feature.

[0107] Specifically, when retrieving the reference image, the first feature in the reference image is extracted based on the first embedding model. And based on the image feature database, according to the first feature, the target image associated with the reference image is retrieved, which realizes the retrieval based on the reference image.

[0108] The reference image and the image used to construct the image feature database can have the same or similar elements, and through feature comparison, the target image associated with the reference image can be retrieved based on the image feature database.

[0109] In one embodiment, the reference image is a certain vehicle part. The image feature database is constructed based on the preset images of a plurality of vehicle parts. Based on the first embedding model, the first feature in the reference image is extracted. And according to the first feature, the target image including the vehicle part belonging to the same type of part or the same type of area as the vehicle part in the reference image is retrieved based on the image feature database.

[0110] Therefore, the image retrieval method of the embodiment of the present application realizes the automatic retrieval of the reference image through the first embedding model and the image feature database, without human intervention, and the retrieval speed is fast and efficient.

[0111] Please refer to Figure 7In some embodiments, step 022, based on the image feature database, the target image associated with the reference image is retrieved according to the first feature, comprising:

[0112] 0221: According to the first feature, the target image feature corresponding to the first feature in the image feature database is determined.

[0113] 0222: According to the target image feature, the target image is determined.

[0114] In some embodiments, the processor can be used to determine the target image feature corresponding to the first feature in the image feature database according to the first feature; and determine the target image according to the target image feature.

[0115] In some embodiments, the retrieval module comprises a first determination submodule and a second determination submodule. The first determination submodule can be used to determine the target image feature corresponding to the first feature in the image feature database according to the first feature; and the second determination submodule can be used to determine the target image according to the target image feature.

[0116] Specifically, in the image feature database, the first image features extracted from a plurality of preset images are stored. The form of the first feature is basically the same as that of the first image feature. According to the first feature, the first image feature corresponding to the first feature is determined. That is, the preset image corresponding to the first image feature can be used as the target image to complete the retrieval of the reference image.

[0117] In one embodiment, the reference image is an image of an automobile part and accessory, the first embedding model is a ViT network model, and the image feature database is a Faiss vector database. The classification layer of the ViT network model is removed, and the last layer is reserved to output a 2048-dimensional feature vector layer for obtaining the first feature of the reference image. That is, the first feature is obtained by extracting a 2048-dimensional feature vector using the first embedding model. Based on the 2048-dimensional first feature, the constructed Faiss vector database is retrieved, and the target image feature of the automobile part and accessory in the Faiss vector database is retrieved based on the HNSW method. Through the target image feature, the original part and accessory images of each automobile brand in the image retrieval folder are matched as the target image, and the output is displayed.

[0118] In this way, by comparing the features, the target image feature corresponding to the first feature of the reference image is retrieved, and then the target image is determined according to the target image feature, so that the reference image is automatically retrieved efficiently and accurately.

[0119] Please refer to Figure 8In some embodiments, the step 0221, according to the first feature, determines a target image feature corresponding to the first feature in the image feature database, including:

[0120] 02211: According to the first feature, the target image feature corresponding to the first feature in the image feature database is determined based on the preset indexing method.

[0121] In some embodiments, the processor can be used to determine the target image feature corresponding to the first feature in the image feature database based on the preset indexing method according to the first feature.

[0122] In some embodiments, the first determination sub-module includes a first determination unit. The first determination unit can be used to determine the target image feature corresponding to the first feature in the image feature database based on the preset indexing method according to the first feature.

[0123] Specifically, the preset indexing method is an indexing method referred to when constructing the image feature database. Based on the preset indexing method, the target image feature corresponding to the first feature is determined according to the first feature, and the retrieval speed is fast. For example, the preset indexing method is a graph-based HNSW approximate nearest neighbor vector retrieval method. This method is a very mainstream and powerful algorithm, with super-fast search speed and excellent recall rate; a hierarchical retrieval method is adopted, and the higher the graph node, the faster the retrieval speed, and the lower the graph node, the slower the retrieval speed. Comprehensive analysis shows that this retrieval scheme can greatly improve the retrieval matching efficiency of image feature vectors.

[0124] It should be noted that the target image feature can be one or more, that is, one or more target image features in the image feature database can be retrieved according to the first feature.

[0125] In one embodiment, the preset indexing method is the HNSW method. Based on the HNSW method, the k image features of the automobile parts and accessories most relevant to the first feature in the Faiss vector database are retrieved as the target image features according to the first feature. Wherein, k is an integer greater than or equal to 1.

[0126] In this way, based on the preset indexing method, the first feature is retrieved in the image feature database, which can speed up the retrieval speed and efficiency.

[0127] Please refer to Figure 9 In some embodiments, the step 02211, based on the preset indexing method, determines the target image feature corresponding to the first feature in the image feature database according to the first feature, including:

[0128] 022111: determining the target image feature based on a preset indexing method according to a similarity between the image feature in the image feature database and the first feature.

[0129] In some embodiments, the processor can be configured to determine the target image feature based on a preset indexing method according to a similarity between the image feature in the image feature database and the first feature.

[0130] In some embodiments, the first determining unit comprises a first determining subunit. The first determining subunit can be configured to determine the target image feature based on a preset indexing method according to a similarity between the image feature in the image feature database and the first feature.

[0131] Specifically, based on the similarity between the first feature and the image feature stored in the image feature database, the target image feature most relevant to the first feature can be determined.

[0132] The determination method of the similarity includes cosine similarity, Euclidean distance, Manhattan distance, Pearson correlation coefficient, etc. The appropriate determination method can be selected according to actual needs to determine the similarity between the first feature and the image feature.

[0133] In one embodiment, the image feature database is automatically constructed based on the image feature embedding model, the similarity between the image features is calculated by cosine similarity, and the HNSW retrieval method is adopted to improve the retrieval efficiency of the system on the database, thereby greatly improving the matching and retrieval accuracy and efficiency of the automobile part image.

[0134] In this way, based on the preset indexing method, the target image feature is determined by the similarity between the first feature and the image feature, and the image feature most relevant to the first feature can be quickly retrieved from the image feature database.

[0135] Please refer to Figure 10 In some embodiments, the image retrieval method further comprises:

[0136] 023: determining the category feature corresponding to the reference image based on the first embedding model, and the category feature is used to determine the reliability of the first feature.

[0137] In some embodiments, the processor can be configured to determine the category feature corresponding to the reference image based on the first embedding model.

[0138] In some embodiments, the image retrieval device comprises a first determining module. The first determining module can be configured to determine the category feature corresponding to the reference image based on the first embedding model.

[0139] Specifically, based on the last classification layer of the first embedding model, the classification of the elements contained in the reference image is realized to determine the category features corresponding to the reference image. The user can judge the category of the elements contained in the reference image by himself, and compare the judgment result with the category features output by the first embedding model, and determine the reliability of the first features output by the first embedding model according to the judgment result and the category features.

[0140] In the case where the judgment result and the category features do not correspond, it can be considered that the first features extracted by the first embedding model are incorrect and cannot accurately express the category of the elements contained in the reference image, and the reliability of the first features is low.

[0141] In the case where the judgment result and the category features correspond, the reliability of the first features is high.

[0142] In this way, the classification auxiliary method is adopted to further assist image matching retrieval, so that the correctness and efficiency of the image retrieval method are further improved, and the method can be applied to various image retrieval fields.

[0143] Please refer to Figure 11 The embodiment of the application provides a training method for training the first embedding model in any of the above embodiments, and the training method comprises the following steps:

[0144] 031: preprocessing the original image to obtain a training image set;

[0145] 032: training a preset model based on a preset optimization method according to the training image set to obtain the first embedding model.

[0146] The embodiment of the application provides an electronic device, which comprises one or more memories and a processor. The memory stores a computer program that can be executed by the processor. The processor can be used to preprocess the original image to obtain a training image set; and train a preset model based on a preset optimization method according to the training image set to obtain the first embedding model.

[0147] The embodiment of the application provides a training device, which comprises a preprocessing module and a training module. The preprocessing module can be used to preprocess the original image to obtain a training image set; and the training module can be used to train a preset model based on a preset optimization method according to the training image set to obtain the first embedding model.

[0148] Specifically, the original images are pre-collected to include multiple images containing target objects. The original images are pre-processed to obtain a training image set, so that the original images can be used to train a preset model. The pre-processing of the original images includes screening, classification, data enhancement, etc. In addition, a validation set and a test set can also be determined according to the pre-processed original images. For example, the original images after pre-processing are divided into a training image set and a validation set.

[0149] In one embodiment, some poor quality data is removed from the collected original images of the region position of the automobile parts by manual operation, and after the parts are classified and arranged according to the specific parts of the automobile and the noise is processed, a high-quality training image set for the training of the embedding model and a validation set for verifying the quality of the embedding model are obtained. The training image set and the validation set are constructed according to a certain rule in a ratio of 4:1.

[0150] The preset optimization method includes an AdaGrad optimization method, an RMSProp optimization method, an Adam Optimizer optimization method, etc. The embodiment is described by taking the Adam Optimizer optimization method as an example.

[0151] The Adam Optimizer optimization method mainly uses a momentum variable v t and an exponentially weighted moving average variable s t of the element square of the small batch random gradient in the RMSProp algorithm:

[0152] v t = β1v t-1 + (1-β1)g t

[0153] s t = β2s t-1 + (1-β2)g t ·g t

[0154]

[0155] wherein β1 and β2 are hyperparameters, representing the exponential decay rate of the first moment estimation and the exponential decay rate of the second moment estimation, respectively. The values of β1 and β2 are set between [0, 1], and the recommended value is 0.9. The initial values v0 and s0 are set to 0, and g t represents the gradient value at time t, is the momentum of the first moment estimation after bias correction at time t, is the weighted translation variable of the second moment estimation after bias correction at time t, and g ′ t is the gradient value after correction at time t.

[0156] In one embodiment, the base model ViT-B of the ViT series is selected as the embedding model, the pre-trained model weight is downloaded, and the adaptive learning rate Adam Optimizer optimization method is used to optimize the embedding model.

[0157] The first-order learning rate decay rate is 0.9, and the second-order learning rate decay rate is 0.999. The batch sample quantity is set to 32, the training round is 100, the image block size is set to 16x16, and the image input size is 224x224. The number of Transformer attention heads is set to 8, the output dimension of the multi-head attention sublayer is 1024, the output dimension of the full-link sublayer is 512, and the number of layers of the Transformer encoder is 12. The two full-link layers of the final classifier are set to 2048 and 1024, the positive sample weight of the loss function is 0.5, and the loss decay rate is 2.0. By loading the pre-trained model weight, using the above hyperparameters, and using the data cleaning and processed automobile parts and accessories training image set, the ViT classification network model is trained, and the validation data set is used to verify the model to identify the optimal model.

[0158] After the training is completed, the ViT-B model is preprocessed, and the classification layer is retained for auxiliary retrieval. The 2048-dimensional feature vector of the second full-link layer of the ViT-B model is the first feature or image feature extracted, which can be used to construct an image feature data set or for image retrieval.

[0159] In this way, by preprocessing the original image, a high-quality training image set can be obtained, so that the inference ability of the model trained based on the training image set is stronger. Training the preset model based on the preset optimization method can accelerate the convergence speed of the model and reduce the time required for training.

[0160] See Figure 12 In some embodiments, 031, the original image is preprocessed to obtain a training image set, including:

[0161] 0311: Preprocessing the original image to obtain a processed image;

[0162] 0312: Classifying the processed image to obtain a training image set.

[0163] In some embodiments, the processor can be configured to: preprocess the original image to obtain a processed image; and classify the processed image to obtain a training image set.

[0164] In some embodiments, the preprocessing module comprises a preprocessing submodule and a classification module. The preprocessing submodule can be used to preprocess the original image to obtain a processed image; and the classification module can be used to classify the processed image to obtain a training image set.

[0165] Specifically, the original image is preprocessed by data cleaning, screening, etc. to obtain a processed image. Then, the processed image is classified according to the characteristics of the target object, and a training image set containing categories is obtained.

[0166] The original image needs to be clean, the target object is in the middle position, and the image is collected at multiple angles. The collected original image is screened to remove unclear and low-quality images, and high-quality images meeting the requirements are retained according to the set rules to obtain the processed image.

[0167] The processed image is classified according to the preset division rule to obtain a training image set. The preset division rule is set according to actual needs.

[0168] In one embodiment, the original image is collected during the automobile exhibition and new car release of each automobile brand. The collected data must be clear and visible, and the center part of the alignment collection component is collected at multiple angles. After preprocessing the collected original image, it is preliminarily classified according to the brand. The area of the automobile part and accessory in the automobile is divided according to the rules of the body exterior area, chassis area, engine cover area, trunk area, automobile interior driving cabin front part area, and automobile interior rear part area. The processed image is sorted and divided according to the above rules, and finally all brand vehicle data is classified into each area.

[0169] In this way, the original image is preprocessed to obtain a processed image, and the processed image is classified to obtain a training image set.

[0170] Please refer to Figure 13 In some embodiments, in step 031 or step 0311, the original image is preprocessed, including:

[0171] 0313: compressing the original image to a preset size to obtain a compressed image;

[0172] 0314: performing normalization processing and / or data enhancement processing on the compressed image to complete the preprocessing of the original image.

[0173] In some embodiments, the processor can be used to: compress the original image to a preset size to obtain a compressed image; and perform normalization processing and / or data enhancement processing on the compressed image to complete the preprocessing of the original image.

[0174] In some embodiments, the preprocessing module comprises a compression submodule and a processing submodule. The compression submodule can be used to compress the original image to a preset size to obtain a compressed image. The processing submodule can be used to normalize and / or data enhance the compressed image to complete the preprocessing of the original image.

[0175] Specifically, to speed up the model training, the size of the original image is compressed to a preset size to obtain a compressed image, so that the preset model is trained according to the compressed image, and the training speed is faster.

[0176] The compressed image is normalized and data enhanced, and the variance and mean of the compressed image are calculated to determine whether the compressed image is balanced. If the balance is poor, the compressed image can be further processed.

[0177] In one embodiment, the 1920x1280 pixel picture originally collected is compressed to 224x224, and the mean and variance of all pictures are calculated to obtain the specific distribution of image data, and the image is normalized and data enhanced to complete the preprocessing of the original image.

[0178] In this way, by compressing, normalizing and data enhancing the original image, the original image can be more suitable for training the model.

[0179] Please refer to Figure 14 In some embodiments, step 032, training the preset model according to the training image set comprises:

[0180] 0321: training the preset model according to the training image set and the focal loss function.

[0181] In some embodiments, the processor can be used to train the preset model according to the training image set and the focal loss function.

[0182] In some embodiments, the training module comprises a training submodule. The training submodule can be used to train the preset model according to the training image set and the focal loss function.

[0183] Specifically, the focal loss function (Focal_loss) is beneficial to the embedding of difficult samples by the model, enhances the distinction of image parts and accessories, enhances the embedding ability of the model, and is beneficial to the construction of the vector library. The formula of the Focal_loss function is:

[0184] Focal_Loss=-α t (1-p t ) γ log(p t )

[0185] wherein the hyper-parameter a t representing the weight of positive samples, parameter p t representing the degree of difficulty of sample classification, the hyper-parameter γ determines the degree of loss attenuation.

[0186] According to the focal loss function, the loss function value of the model inference result is calculated, and the preset model is adjusted according to the loss function value, so as to realize the training of the preset model.

[0187] In this way, the preset model is trained by using the training image set, the training effect of the preset model is evaluated by using the focal loss function, and the parameters of the preset model are further adjusted according to the evaluation result, so as to realize the further adjustment and training of the preset model.

[0188] In some embodiments, the original image satisfies at least one of the preset proportion condition, the preset angle condition, the preset light condition, and the preset quality condition.

[0189] Specifically, according to at least one of the preset proportion condition, the preset angle condition, the preset light condition, and the preset quality condition, the original image is collected, so as to ensure the quality of the original image.

[0190] The preset proportion condition means that the proportion of the target object in the original image needs to be within a preset proportion range.

[0191] The preset angle condition means that the original image needs to include images taken at multiple set angles. For example, please refer to Figure 15 The preset angle is defined as (φ, r, θ), and data collection can be performed according to eight angles of (0°, φ(δ), 45°), (45°, φ(δ), 45°), (90°, φ(δ), 45°), (135°, φ(δ), 45°), (180°, φ(δ), 45°), (225°, φ(δ), 45°), (270°, φ(δ), 45°), (315°, φ(δ), 45°). Among them, φ(δ) is determined according to the proportion δ of the target object in the original image.

[0192] The preset light adjustment means that the original image needs to be collected under sufficient light conditions when taking the original image, so that the original image can clearly show every detail feature of the image. The preset quality adjustment means that the resolution of the original image is within a preset resolution range.

[0193] In one embodiment, the mobile device is used to collect vehicle data under certain preset proportion conditions, preset angle conditions, preset lighting conditions, and preset quality conditions to collect target objects, ensuring the diversity and richness of data distribution, which helps to enhance the robustness of the model and improve the accuracy of regional modeling recognition. Since the image acquisition quality requirement is high for vehicle competitive product analysis image comparison, and each detail feature of the automobile parts and accessories needs to be highlighted, the original image needs to be taken under sufficient light adjustment.

[0194] In this way, at least one of the preset proportion conditions, the preset angle conditions, the preset lighting conditions, and the preset quality conditions is set for the original image

[0195] In some embodiments, the preset proportion conditions include that the proportion of the area occupied by the target object in the original image is within a preset proportion range; and / or, the preset quality conditions include that the resolution of the original image is a preset resolution.

[0196] Specifically, the proportion of the area occupied by the target object in the original image in the whole original image is defined as the measured distance of the target object. If the length of the original image is h and the width is w; the length of the target object is h ′ , and the width is w ′ . The proportion δ of the target object is calculated.

[0197]

[0198] The preset proportion range is a preset proportion range value, which can be set according to actual needs. For example, in order to obtain the detailed information of automobile parts and accessories, the target object must occupy most of the area of the collection device, and therefore the threshold space range for the proportion δ is set to [0.7, 0.95].

[0199] The preset quality condition refers to the resolution requirement of the original image. For example, in order to ensure the high quality of the image and the obviousness of the detailed features, the resolution requirement of the image is 1920x1280 with high degree of features.

[0200] In one embodiment, the original image is obtained by taking pictures at public occasions or using a web crawler. The collected original image is cleaned, and the data of each brand is divided according to the vehicle area, and then the types of parts in each area are classified and processed to construct the training image set and the verification set of the first embedding model of the image feature database.

[0201] The ViT classification algorithm model is used as the first embedding model, and the pre-training model weight corresponding to the model is downloaded. The training image set constructed by the collected parts of various brands of automobiles is used to pre-tune the ViT model, and the test set is used to test the model to reserve the best model. The last classification layer of the ViT is removed, and the last feature layer with an output dimension of 2048 dimensions is reserved to output the image features of the corresponding preset image.

[0202] The Faiss vector database is used as the storage medium of the image features, and all the collected preset images are embedded into the Faiss vector database through the ViT embedding model for storage. The cosine similarity is used to measure the similarity between two image feature vectors for the image feature retrieval of the Faiss vector database, and the graph-based approximate nearest neighbor vector retrieval method HNSW is used for retrieval. In order to further improve the accuracy of retrieval, the category filtering is performed before the final similarity retrieval, and the same features as the first feature category are compared, so as to further improve the accuracy and efficiency of the retrieval.

[0203] Finally, the images with top-k similarity to the automobile parts and accessories to be matched are selected for display and comparison, which assists the automobile competitive product analysis personnel in analyzing the competitive products of various automobile parts and accessories, and the product design personnel in comparing and analyzing the detailed features of various brand products.

[0204] The embodiment of the present application provides a computer readable storage medium, which stores a computer program. When the program is executed by a processor, the steps of the method of any of the above embodiments are implemented.

[0205] The embodiment of the present application provides a computer program product, which comprises a computer program. When the computer program is executed by a processor, the steps of any of the above embodiments are implemented.

[0206] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms is not necessarily directed to the same embodiment or example. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0207] In addition, the term "connection" should be interpreted broadly, for example, it can include fixed connection, or detachable connection, or integral connection; it can include direct connection, or indirect connection through intermediate medium, or communication between two elements inside. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0208] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0209] Any process or method descriptions, flow diagrams, or other descriptions of processes or methods described herein or in the flow diagrams can be understood as representing modules, segments, or portions of code that include one or more executable instructions for implementing specific logical functions or steps in the process, and that the scope of preferred embodiments of the present application encompasses alterations, modifications, and variations of these processes or methods that can be made by those skilled in the art, in which the order of steps can be changed, including according to the functions involved, in substantially simultaneous manner or in reverse order, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0210] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for constructing an image feature database, characterized in that, The construction method includes: Based on the pre-trained first embedding model, extract the first image features of the preset image; Based on the first image features, construct the image feature database.

2. The construction method according to claim 1, characterized in that, The construction method also includes: Based on a preset indexing method, a preset image feature database is constructed; The step of constructing the image feature database based on the first image feature includes: The image feature database is constructed based on the first image feature and the preset image feature database.

3. The construction method according to claim 2, characterized in that, The construction method also includes: The database environment for constructing the image feature database; In the database environment, the preset image feature database is initialized according to preset storage hyperparameters.

4. The construction method according to claim 1, characterized in that, The construction method also includes: Establish a feature link between the preset image and the features of the first image, and the feature link can be used to determine the corresponding preset image based on the features of the first image.

5. An image retrieval method, characterized in that, The image retrieval method is based on the image feature database constructed using the construction method described in any one of claims 1-4, and the image retrieval method includes: Based on the first embedding model, the first feature is extracted from the reference image; Based on the image feature database, a target image associated with the reference image is retrieved according to the first feature.

6. The image retrieval method according to claim 5, characterized in that, The step of retrieving a target image associated with the reference image based on the image feature database and according to the first feature includes: Based on the first feature, determine the target image feature corresponding to the first feature in the image feature database; The target image is determined based on the characteristics of the target image.

7. The image retrieval method according to claim 6, characterized in that, The step of determining the target image feature corresponding to the first feature in the image feature database based on the first feature includes: Based on a preset indexing method, the target image feature corresponding to the first feature in the image feature database is determined according to the first feature.

8. The image retrieval method according to claim 7, characterized in that, The method based on a preset index, which determines the target image feature corresponding to the first feature in the image feature database according to the first feature, includes: Based on a preset indexing method, the target image features are determined according to the similarity between the image features in the image feature database and the first feature.

9. The image retrieval method according to claim 5, characterized in that, The image retrieval method further includes: Based on the first embedding model, the category features corresponding to the reference image are determined, and the category features are used to determine the reliability of the first feature.

10. A training method, characterized in that, The training method for training the first embedding model as described in any one of claims 1-9 includes: The original images are preprocessed to obtain a training image set; Based on a preset optimization method, the preset model is trained according to the training image set to obtain the first embedding model.

11. The training method according to claim 10, characterized in that, The preprocessing of the original images to obtain the training image set includes: The original image is preprocessed to obtain the processed image; The processed images are classified to obtain the training image set.

12. The training method according to claim 10 or 11, characterized in that, The preprocessing of the original image includes: The original image is compressed to a preset size to obtain a compressed image; The compressed image is subjected to normalization and / or data augmentation to complete the preprocessing of the original image.

13. The training method according to claim 10, characterized in that, The step of training the preset model based on the training image set includes: The preset model is trained based on the training image set and the focus loss function.

14. The training method according to claim 10, characterized in that, The original image satisfies at least one of the following preset conditions: percentage ratio, angle, lighting, and quality.

15. The training method according to claim 14, characterized in that, The preset proportion condition includes: the proportion of the area occupied by the target object in the original image is within the preset proportion range; and / or, The preset quality conditions include: the resolution of the original image is a preset resolution.

16. An electronic device, characterized in that, The electronic device includes one or more processors and a memory, the memory storing a computer program that, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 15.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 15.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images