Design patent image retrieval method and apparatus, device and medium

Abstract

Provided in the embodiments of the present description are a design patent image retrieval method and apparatus, a device and a medium. The method comprises: constructing a data set consisting of data pairs consisting of design images and text labels representing attributes of the design images; inputting the data set into an initial Clip model for training same; then inputting the data set into the model to generate first feature vectors and second feature vectors corresponding to text features, and on the basis of a preset vector weighting proportion, calculating respective fusion feature vectors; clustering the fusion feature vectors on the basis of a clustering algorithm to obtain sub-data sets, the sub-data sets storing fusion feature vectors, or the sub-data sets storing the first feature vectors of image features; and inputting an image to be retrieved into an image-text encoder of the trained Clip model to obtain an image feature vector, so as to obtain a retrieval image result. The present invention solves the problems of inaccurate categories of retrieval and time consumption of retrieval.

Description

A method, device, equipment and medium for searching appearance patent images Technical Field

[0001] The present invention relates to the field of image analysis technology, and in particular to a method, device, equipment and medium for searching appearance patent images. Background Art

[0002] Image retrieval can be divided into two categories according to the different ways of describing image content: one is text-based image retrieval (TBIR) and the other is content-based image retrieval (CBIR).

[0003] Among them, text-based image retrieval (TBIR) requires manual annotation of text keywords corresponding to images. By establishing associations between images and text keywords, the image retrieval problem is successfully transformed into a text keyword retrieval problem. However, manual annotation of the correspondence between images and keywords is time-consuming and costly for large-scale image datasets. Furthermore, because images and their corresponding keywords are annotated one by one, these image-text pairs are isolated from each other, making it impossible for the retrieval system to jointly learn the concepts in the text keywords, resulting in inaccurate retrieval.

[0004] CBIR is currently the mainstream technology for image retrieval. It directly uses images as search criteria, achieving true image-by-image search. This technology is more advanced than TBIR and avoids the semantic misalignment problem that occurs when indirectly retrieving images through text. CBIR also primarily uses underlying visual features such as color, texture, and shape to represent images. Compared to TBIR, it can capture more detailed image features, significantly improving its accuracy and gaining widespread use. However, it has the following limitations:

[0005] (1) The retrieval results are easily affected by factors such as the shooting angle, lighting intensity, occlusion, and deformation degree of the image. Many images cannot be distinguished from their specific content, especially some abstract product images.

[0006] (2) This technology uses the underlying visual features of the image, such as color, texture, and shape, to encode the image, while ignoring higher-level semantic information. For example, objects with similar color, texture, and shape may be products of two completely different categories. Therefore, the Locarno classification cannot be simply used to directly classify the appearance patent images, resulting in inaccurate retrieval categories and time-consuming retrieval.

[0007] In view of this, there is an urgent need to provide a method for retrieving appearance patent images with higher retrieval accuracy and faster speed.

[0008] Summary of the Invention

[0009] In order to overcome the problems existing in the related art, the present disclosure provides a method, device, equipment and medium for retrieval of appearance patent images to solve the technical problems of inaccurate appearance image retrieval categories and time-consuming retrieval in the related art.

[0010] One or more embodiments of this specification provide a method for searching appearance patent images, including the following steps:

[0011] Construct a dataset consisting of appearance images and text label data pairs representing the attributes of the appearance images;

[0012] Input the dataset into the initial Clip (Contrastive language-image pre-training) model to train it to obtain the trained Clip model's image-text encoder;

[0013] The dataset is then input into the image-text encoder of the trained Clip model to generate a first feature vector corresponding to the appearance image features and a second feature vector corresponding to the text features. The first feature vector and the second feature vector are weighted and calculated according to a preset vector weight ratio to obtain each fused feature vector.

[0014] Clustering the fused feature vectors based on a clustering algorithm to obtain sub-datasets, storing the fused feature vectors in each sub-dataset, and calculating the average value of all the fused feature vectors in each sub-dataset as the first category feature vector of the corresponding sub-dataset; or storing the first feature vector of the image feature in each sub-dataset, and calculating the average value of all the first feature vectors in each sub-dataset as the second category feature vector of the corresponding sub-dataset;

[0015] The image to be retrieved is input into the image-text encoder of the trained Clip model to obtain an image feature vector. The similarity between the image feature vector and the category vector of each sub-dataset is calculated to obtain the top n target sub-datasets. The similarity between the image feature vector and each target sub-dataset is then calculated to obtain the top m target image feature vectors, and the retrieved image results are obtained.

[0016] Furthermore, the Clip model calculates a fused feature vector according to set weights, wherein the weights of the first feature vector and the second feature vector are set to 0.7 and 0.3, respectively.

[0017] Furthermore, the sub-dataset stores the fused feature vector or the first feature vector corresponding to the image feature, and the feature vectors contained in the sub-dataset are indexed and stored through the faiss library.

[0018] Furthermore, when the sub-dataset stores the first feature vector of the image feature, and when the average value of all the first feature vectors in the sub-dataset is calculated as the second category feature vector of the corresponding sub-dataset, the retrieval process includes:

[0019] Extract features from an input image to be retrieved using the image-text encoder of the Clip model to obtain an image feature vector, perform similarity calculations between the obtained image feature vector and each second category vector of each first sub-dataset, and obtain the top n ranked target sub-datasets based on a preset first threshold; and

[0020] Then, the target image feature vectors are retrieved through the index files of each target sub-dataset, and the similarity between the image feature vector and the target image feature vector corresponding to each index is calculated. According to the preset second threshold, the top m target image feature vectors in the calculation results are selected and sorted, and finally the images corresponding to n×m target image feature vectors are obtained.

[0021] Furthermore, when the fused feature vectors are stored in the sub-dataset, and when the average value of all the fused feature vectors in the sub-dataset is calculated as the first category feature vector of the corresponding sub-dataset, the retrieval process includes:

[0022] When the image to be retrieved and the text information corresponding to the image are input, features are extracted through the image-text encoder of the Clip model to obtain image feature vectors and text feature vectors, and a fused feature vector is calculated based on the preset weight ratio of the image feature vector and the text feature vector; the similarity between the fused feature vector and the first category vector of each second sub-data set is calculated, and according to the preset third threshold, the top n target sub-data sets are obtained, and then the target image feature vectors are retrieved through the index files of each target sub-data set, and the similarity between the fused feature vector and the fused feature vector corresponding to the index in each second sub-data set is calculated, and according to the preset fourth threshold, the top m target image feature vectors in the calculation results are selected, and finally the images corresponding to the n×m target image feature vectors are obtained.

[0023] One or more embodiments of this specification provide a device for searching appearance patent images, including:

[0024] A dataset construction module is used to construct a dataset consisting of appearance images and text label data pairs representing the attributes of the appearance images;

[0025] The model training module is used to input the data set into the initial Clip model to train it, so as to obtain the image-text encoder of the trained Clip model;

[0026] A feature vector acquisition module is used to input the dataset into the image-text encoder of the trained Clip model to generate a first feature vector corresponding to the appearance image features and a second feature vector corresponding to the text features, and to calculate the weighted sum of the first feature vector and the second feature vector according to a preset vector weight ratio to obtain a fused feature vector;

[0027] a feature clustering module, configured to cluster the fused feature vectors based on a clustering algorithm to obtain a sub-dataset, wherein the sub-dataset stores the fused feature vectors and calculates the average value of all the fused feature vectors in the sub-dataset as the first category feature vector of the sub-dataset; or stores the first feature vector of the image feature in the sub-dataset and calculates the average value of all the first feature vectors in the sub-dataset as the second category feature vector of the sub-dataset; and

[0028] The retrieval module is used to input the image to be retrieved into the image-text encoder of the trained Clip model to obtain an image feature vector, perform similarity calculation between the image feature vector and the category vector of the sub-dataset to obtain the top n target sub-datasets, and then perform similarity calculation between the image feature vector and each feature vector stored in the target sub-dataset to obtain the top m target image feature vectors, thereby obtaining the retrieved image results, wherein n is a natural number greater than 1, and m is a natural number less than or equal to n.

[0029] Furthermore, it also includes a weight setting module for setting the dimensions and weights of the first eigenvector and the second eigenvector, wherein the weights of the first eigenvector and the second eigenvector are set to 0.7 and 0.3 respectively.

[0030] Furthermore, the sub-dataset stores the fused feature vector or the first feature vector corresponding to the image feature, and the feature vectors contained in the sub-dataset are indexed and stored through the faiss library.

[0031] One or more embodiments of this specification provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned appearance patent image retrieval methods is implemented.

[0032] One or more embodiments of this specification provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned appearance patent image retrieval methods.

[0033] The present disclosure provides a method, device, equipment and medium for searching appearance patent images. The advantages are that, compared with the existing method using underlying image features such as color, texture, and shape of an image, the present method enables a visual language model to learn both visual features and high-level text semantics through comparative learning, and aligns the two features in the same feature space, so that the model can learn the connection between image and text concepts, making the model retrieval more accurate. The method innovatively classifies the images reasonably and uses the classified image subsets as retrieval comparison objects, which can further avoid semantically similar products of different categories from being retrieved together, thereby ensuring the accuracy of the retrieval categories and results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] FIG1 is a flow chart of a method for searching appearance patent images provided by one or more embodiments of this specification;

[0036] FIG2 is a flowchart of the Clip model training, image-text fusion feature data set, and category feature vector generation process provided by one or more embodiments of this specification;

[0037] FIG3 is a flowchart of a process for retrieving an image according to one or more embodiments of the present specification;

[0038] FIG4 is a schematic diagram of the first module composition of a device for searching appearance patent images provided by one or more embodiments of this specification;

[0039] FIG5 is a schematic diagram of the structure of a computer provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0040] In order to help those skilled in the art better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this invention.

[0041] The present invention will be described in detail below with reference to specific implementation methods and the accompanying drawings.

[0042] Method Example

[0043] According to an embodiment of the present invention, a method for retrieving appearance patent images is provided. FIG1 is a flow chart of the method for retrieving appearance patent images provided in this embodiment. The method for retrieving appearance patent images according to an embodiment of the present invention includes:

[0044] Step S1: Construct a dataset consisting of appearance images and text label data pairs representing appearance image attributes (categories). The text labels of appearance image attributes are used to represent image attributes, such as clothing, dining table, toys, etc.

[0045] Step S2: Input the data set into the initial Clip (Contrastive Language-Image Pre-Training) model for training to obtain a trained Clip model image-text encoder.

[0046] In this embodiment, reference may be made to FIG2 , which is a flowchart of the Clip model training, image-text fusion feature data set, and category feature vector generation process provided in this embodiment. The image-text encoder of the Clip model is a large visual language model. The model is fine-tuned by using the appearance patent image and the name text data pair corresponding to the image to form an image-text data pair data set. In this embodiment, the image visual features of the image-text data pair and the semantic features of the text are mapped to the same feature space through comparative learning of the large visual language model. In this way, the model can establish a connection between the image and text semantic concepts. The model can not only learn the underlying visual features of the image, but also establish a connection with the advanced text semantic features. Moreover, the model constructs a concept set for all semantic features in the data set, rather than isolating the image and text information from each other. This allows the trained Clip model to flexibly meet image retrieval of text, text retrieval of image, or mixed retrieval of image-text fusion features, so that the retrieval system can more flexibly meet the needs of users and the model retrieval is more accurate.

[0047] In this embodiment, the Clip visual language model is introduced as the encoding model for data pairs, wherein the visual feature encoder uses the transformer-based ViT (Visual Transformer) model, and the text semantic encoder uses the RoBERTa model, which are used to improve the representation ability of image features and text semantic features; and the feature vector extracted by the visual language model contains both visual features and text semantic features. The model can flexibly meet the needs of image retrieval of text, text retrieval of image, or mixed retrieval of image and text fusion features, so that the retrieval system can more flexibly meet the needs of users.

[0048] Step S3: input the data set into the image-text encoder of the trained Clip model to generate a first feature vector corresponding to the appearance image feature and a second feature vector corresponding to the text feature, and calculate the weighted sum of the first feature vector and the second feature vector according to a preset vector weight ratio to obtain a fused feature vector.

[0049] In this embodiment, the dimensions and weights of the first feature vector and the second feature vector are set according to the degree of influence of the image features and text features on the retrieval accuracy, and the Clip model obtains the fused feature vector based on the set weight calculation; in this embodiment, the weights of the first feature vector and the second feature vector are set to 0.7 and 0.3 respectively. Assuming that the first feature vector and the second feature vector are both 512, the fused feature vector = first feature vector × 0.7 + second feature vector × 0.3.

[0050] Step S4: clustering the fused feature vectors based on a clustering algorithm to obtain a sub-dataset, and calculating the average value of all the fused feature vectors in the sub-dataset as the first category feature vector of the sub-dataset;

[0051] Or the first feature vector of the image feature is stored in the sub-dataset, and the average value of all the first feature vectors in the sub-dataset is calculated as the second category feature vector of the sub-dataset.

[0052] In one embodiment, referring to FIG2 , the fused feature vector set is clustered using a K-means clustering algorithm according to a preset K value. For example, the entire data set is clustered into 100 categories, and then the mean of the first feature vectors of all fused feature vectors or all image features contained in each category is used to represent the category vector of the corresponding category; the details are as follows.

[0053] In one embodiment, considering the convenience of retrieval and the usage of retrieval scenarios, a general user retrieves appearance patents by inputting the image to be retrieved. Therefore, after the clusters are obtained after clustering, the first feature vectors corresponding to the images contained in each cluster are stored to form a corresponding first sub-dataset, and then the mean of the first feature vectors contained in each first sub-dataset is used as the corresponding first sub-dataset category vector for retrieval.

[0054] In another embodiment, in order to meet the needs of higher-precision retrieval, this embodiment also uses the fused feature vector as a storage object and saves it as a second sub-dataset after clustering, and then uses the mean of the fused feature vector contained in each second sub-dataset as the corresponding second sub-dataset category vector for retrieval; that is, when the retrieval object is a picture and text information, the Clip model will extract the features of the picture and text respectively and convert them into corresponding feature vectors, and then calculate to obtain the fused features, and match the fused features with the second sub-dataset to determine the retrieval result image.

[0055] In this embodiment, the weight of the first eigenvector representing the image feature is set to be greater than the second eigenvector representing the text feature. This is because during the retrieval process, users usually retrieve the appearance by inputting an image. Therefore, the weights of the first eigenvector and the second eigenvector are set to 0.7 and 0.3 respectively to ensure high category accuracy after clustering and high accuracy of the retrieval results.

[0056] In this embodiment, in order to improve the retrieval efficiency, an index file is created for each sub-data set and stored in the index library. Preferably, the faiss library is used to construct an index for the image feature vector contained in the first sub-data set of each category and the fusion feature vector contained in the second sub-data set. The index uses the HNSW approximate nearest neighbor algorithm. During the retrieval process, the retrieval efficiency can be greatly improved through index retrieval.

[0057] In this embodiment, the image sets of each category obtained by the above-mentioned classification method considering image features and text features can further avoid the appearance images with similar image features but different attributes or categories from being retrieved together. This is because the appearance patent images are special. For example, a product can include design drawings from multiple perspectives. It is difficult to tell what the product is from the appearance of some perspectives, that is, the problem of not looking alike. In addition, the appearance patent designs of the same product can be very different from each other. Even some designs cannot be manually classified. There are more than one million product categories. For example, a round-mouthed white ceramic cup with water and some round-mouthed ceramic urinals with water can be easily distinguished if only image features are used for identification. It is easy to be misjudged as the same type of goods, and the two should not be retrieved at the same time in terms of category. Therefore, compared with the above-mentioned image library constructed by simply using the Locarno classification to divide the appearance patent images, the classification accuracy of each type of image set itself is very high; and because the image-text encoder of the Clip model in this embodiment is trained through image-text data pairs, the image-text encoder has very accurate image recognition ability during the retrieval stage, so the obtained feature vector representation ability is also very high. The obtained feature vector is then used to retrieve the target image with the category vector and index library of each sub-data set, which effectively avoids the appearance of incorrect categories in the retrieval results.

[0058] Step S5: Input the image to be retrieved into the image-text encoder of the trained Clip model to obtain an image feature vector, perform similarity calculation between the image feature vector and the category vector of the sub-dataset to obtain the top n target sub-datasets, and then perform similarity calculation between the image feature vector and the target sub-dataset to obtain the top m target image feature vectors, thereby obtaining the retrieved image results, wherein n is a natural number greater than 1, and m is a natural number less than or equal to n. In this embodiment, preferably, reference is made to FIG3, which is a flowchart of the image retrieval process provided by this embodiment. Specifically,

[0059] According to the input image to be retrieved, the image feature vector is obtained by extracting features through the image-text encoder of the Clip model. Since the input is an image, the Clip model only uses the trained image feature extraction model to extract image features and convert them into vectors. When the similarity between the image feature vector and the second category vector of each first sub-dataset is calculated, and according to the preset threshold, the top n target sub-datasets are obtained, and then the target image feature vectors are retrieved through the index files of each target sub-dataset, and the similarity between the image feature vector and the target image feature vector corresponding to each index is calculated, and according to the preset threshold, the top m target image feature vectors in the calculation result are selected, and finally the images corresponding to the n×m target image feature vectors are obtained.

[0060] In another embodiment, when the input to be retrieved includes not only an image but also text information corresponding to the image, features are extracted by the image-text encoder of the Clip model to obtain image feature vectors and text feature vectors, and then a fused feature vector is obtained based on a preset weight ratio of the image feature vector and the text feature vector. Then, similarity is calculated between the fused feature vector and the first category vector of each second sub-data set, and based on a preset threshold, the top n target sub-data sets are obtained. Then, the target image feature vector is retrieved through the index file of each target sub-data set, and similarity is calculated between the fused feature vector and the fused feature vector corresponding to the index in each second sub-data set. Based on the preset threshold, the top m target image feature vectors in the calculation result are selected, and finally images corresponding to n×m target image feature vectors are obtained.

[0061] Preferably, in this embodiment, the n×m target images obtained through the above retrieval will be sorted from large to small according to the calculated similarity values, and the retrieval results will be displayed according to the sorting order. Each image obtained by the retrieval includes information such as the image, similarity value, classification number and a brief description of the appearance image.

[0062] Compared with the existing methods that use low-level image features such as color, texture, and shape of images, the method of the present invention uses comparative learning to enable the visual language model to learn both visual features and high-level text semantics, and align the two features in the same feature space, so that the model can learn the connection between image and text concepts, making model retrieval more accurate. It innovatively classifies images reasonably and uses the classified image subsets as retrieval comparison objects, which can further avoid semantically similar products of different categories from being retrieved together, thereby ensuring the accuracy of retrieval categories and results.

[0063] Device embodiment

[0064] According to an embodiment of the present invention, a device for searching appearance patent images is provided. FIG4 is a schematic block diagram of the device for searching appearance patent images provided in this embodiment. The device for searching appearance patent images according to an embodiment of the present invention includes:

[0065] The dataset construction module 10 is used to construct a dataset consisting of appearance images and text label data pairs representing the attributes of the appearance images. The text labels of the appearance image attributes are used to represent the attributes of the image, such as clothes, dining table, toys, etc.

[0066] The model training module 20 is used to input the data set into the initial Clip model to train it, so as to obtain a graphic encoder of the trained Clip model.

[0067] In this embodiment, the image-text encoder of the Clip model is a large visual language model. The model is fine-tuned by using the appearance patent image and the name text data pair corresponding to the image to form a data set. In this embodiment, the image visual features of the image-text data pair and the semantic features of the text are mapped to the same feature space through comparative learning of the large visual language model. In this way, the model can establish a connection between the image and text semantic concepts. The model can not only learn the underlying visual features of the image, but also establish a connection with the high-level text semantic features. Moreover, the model constructs a concept set for all semantic features in the data set, rather than isolating the image and text information from each other. The trained Clip model can flexibly meet the image retrieval text, text retrieval image or image and text fusion feature mixed retrieval, so that the retrieval system can more flexibly meet the needs of users and the model retrieval is more accurate.

[0068] In this embodiment, the Clip visual language model is introduced as the encoding model of the data pair, where the visual feature encoder uses the transformer-based ViT model, and the text semantic encoder uses the RoBERTa model to improve the representation ability of image features and text semantic features.

[0069] The feature vector acquisition module 30 is used to input the data set into the image-text encoder of the trained Clip model to generate a first feature vector corresponding to the appearance image feature and a second feature vector corresponding to the text feature, and calculate the weighted sum of the first feature vector and the second feature vector according to a preset vector weight ratio to obtain a fused feature vector.

[0070] In this embodiment, a weight setting module 40 is further included, which is used to set the dimensions and weights of the first feature vector and the second feature vector according to the degree of influence of the image features and text features on the retrieval accuracy. The Clip model obtains the fused feature vector based on the set weight calculation; in this embodiment, the weights of the first feature vector and the second feature vector are set to 0.7 and 0.3 respectively. Assuming that the first feature vector and the second feature vector are both 512, the fused feature vector = first feature vector × 0.7 + second feature vector × 0.3.

[0071] A feature clustering module 50 is configured to cluster the fused feature vectors based on a clustering algorithm to obtain a sub-dataset, and calculate an average value of all the fused feature vectors in the sub-dataset as a first category feature vector of the corresponding sub-dataset;

[0072] Or the first feature vector of the image feature is stored in the sub-dataset, and the average value of all the first feature vectors in the sub-dataset is calculated as the second category feature vector of the corresponding sub-dataset.

[0073] In one embodiment, the fused feature vector set is clustered using a K-means clustering algorithm according to a preset K value. For example, the entire data set is clustered into 100 categories, and then the category vector of the corresponding category is represented by the mean of all fused feature vectors or the first feature vectors of all image features contained in each category. The clustering process of the feature clustering module 50 includes the following scenarios:

[0074] In one embodiment, considering the convenience of retrieval and the usage of retrieval scenarios, a general user retrieves appearance patents by inputting the image to be retrieved. Therefore, after the clusters are obtained after clustering, the first feature vectors corresponding to the images contained in each cluster are stored to form a corresponding first sub-dataset, and then the mean of the first feature vectors contained in each first sub-dataset is used as the corresponding first sub-dataset category vector for retrieval.

[0075] In another embodiment, in order to meet the needs of higher-precision retrieval, this embodiment also uses the fused feature vector as a storage object and saves it as a second sub-dataset after clustering, and then uses the mean of the fused feature vector contained in each second sub-dataset as the corresponding second sub-dataset category vector for retrieval; that is, when the retrieval object is a picture and text information, the Clip model will extract the features of the picture and text respectively and convert them into corresponding feature vectors, and then calculate to obtain the fused features, and match the fused features with the second sub-dataset to determine the retrieval result image.

[0076] In this embodiment, in order to improve the retrieval efficiency, an index file is created for each sub-data set and stored in the index library. Preferably, the faiss library is used to construct an index for the image feature vector contained in the first sub-data set of each category and the fusion feature vector contained in the second sub-data set. The index uses the HNSW approximate nearest neighbor algorithm. During the retrieval process, the retrieval efficiency can be greatly improved through index retrieval.

[0077] In this embodiment, the weight of the first eigenvector representing the image feature is set to be greater than the second eigenvector representing the text feature. This is because during the retrieval process, users usually retrieve the appearance by inputting an image. Therefore, the weights of the first eigenvector and the second eigenvector are set to 0.7 and 0.3 respectively to ensure that the category accuracy obtained after clustering by the feature clustering module 50 is high, and at the same time, the accuracy of the retrieval results is high.

[0078] The retrieval module 60 is used to input the image to be retrieved into the image-text encoder of the trained Clip model to obtain an image feature vector, and perform similarity calculation between the image feature vector and the category vector of each sub-dataset to obtain the top n target sub-datasets, and then perform similarity calculation between the image feature vector and each target sub-dataset to obtain the top m target image feature vectors, thereby obtaining the retrieved image result.

[0079] In this embodiment, the search module 60 preferably includes the following two situations:

[0080] When the input is an image, the image feature vector is obtained by extracting features through the Clip model's image-text encoder. Since the input is an image, the Clip model only uses the trained image feature extraction model to extract image features and convert them into vectors. When the image feature vector is calculated with the second category vector of each first sub-dataset similarity, and according to the preset threshold, the top n target sub-datasets are obtained, and then the target image feature vectors are retrieved through the index files of each target sub-dataset, and the similarity between the image feature vector and the target image feature vector corresponding to each index is calculated, and according to the preset threshold, the top m target image feature vectors in the calculation result are selected, and finally the image corresponding to the n×m target image feature vectors is obtained.

[0081] In another embodiment, when the input to be retrieved includes not only an image but also text information corresponding to the image, features are extracted by the image-text encoder of the Clip model to obtain image feature vectors and text feature vectors, and a fused feature vector is obtained according to a preset weight ratio. The similarity between the fused feature vector and the first category vector of each second sub-data set is calculated, and according to a preset threshold, the top n target sub-data sets are obtained. The target image feature vectors are then retrieved through the index files of each target sub-data set, and the similarity between the fused feature vector and the fused feature vector corresponding to the index in each second sub-data set is calculated. According to the preset threshold, the top m target image feature vectors in the calculation results are selected, and finally the images corresponding to the n×m target image feature vectors are obtained.

[0082] Preferably, in this embodiment, the n×m target images obtained by the above-mentioned retrieval module 60 will also be sorted from large to small according to the calculated similarity values, and the retrieval results will be displayed according to the sorting order. Each image obtained by retrieval includes information such as the image, similarity value, classification number and brief description of the appearance image.

[0083] The device of this embodiment introduces a large visual language model and uses the appearance patent image and product name text to form an image-text data pair data set to achieve model training and fine-tuning of the model accuracy. The model maps the image visual features of the image-text data pair and the semantic features of the text to the same feature space through comparative learning. In this way, the model can establish a connection between the image and text semantic concepts. The model can not only learn the underlying visual features of the image, but also establish a connection with the high-level text semantic features. Moreover, the model constructs a concept set for all semantic features in the data set, rather than isolating the data pairs from each other. The visual language model fine-tuned on the appearance image-text data pair set can greatly improve the representation ability of the appearance patent by constructing a multimodal feature vector, and can greatly improve the accuracy of image retrieval without the need for manual annotation intervention.

[0084] The embodiment of the present invention is an apparatus embodiment corresponding to the above-mentioned method embodiment. The specific operations of the processing steps of each module can be understood by referring to the description of the method embodiment, and will not be repeated here.

[0085] As shown in FIG5 , the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for retrieving appearance patent images in the above-mentioned embodiment is implemented. Alternatively, when the computer program is executed by a processor, the method for retrieving appearance patent images in the above-mentioned embodiment is implemented.

[0086] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0087] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and the contents not described in detail in the specification of the present invention are common knowledge to those skilled in the art.

Claims

1. A method for retrieving appearance patent images, characterized in that include: Construct a dataset consisting of appearance images and text label data pairs representing the attributes of the appearance images; Input the data set into the initial Clip model to train it, so as to obtain the image-text encoder of the trained Clip model; The data set is then input into the image-text encoder of the trained Clip model to generate a first feature vector corresponding to the appearance image feature and a second feature vector corresponding to the text feature, and the first feature vector and the second feature vector are weighted and calculated according to a preset vector weight ratio to obtain a fused feature vector; Clustering the fused feature vectors based on a clustering algorithm to obtain a sub-dataset, storing the fused feature vectors in the sub-dataset, and calculating the average value of all the fused feature vectors in the sub-dataset as the first category feature vector of the sub-dataset; or storing the first feature vector of the image feature in the sub-dataset, and calculating the average value of all the first feature vectors in the sub-dataset as the second category feature vector of the sub-dataset; as well as The image to be retrieved is input into the image-text encoder of the trained Clip model to obtain an image feature vector, and similarity calculation is performed between the image feature vector and the category vector of the sub-dataset to obtain the top n target sub-datasets, and then similarity calculation is performed between the image feature vector and each feature vector stored in the target sub-dataset to obtain the top m target image feature vectors, thereby obtaining the retrieved image results, wherein n is a natural number greater than 1, and m is a natural number less than or equal to n.

2. The design patent image retrieval method according to claim 1, characterized in that: The Clip model obtains a fused feature vector according to set weight calculation, wherein the weights of the first feature vector and the second feature vector are set to 0.7 and 0.3 respectively.

3. The design patent image retrieval method according to claim 1, characterized in that: The sub-dataset stores the fused feature vector or the first feature vector corresponding to the image feature, and indexes the feature vectors contained in the sub-dataset through the faiss library and stores them.

4. The design patent image retrieval method according to claim 3, characterized in that: When the first feature vector of the image feature is stored in the sub-dataset, and when the average value of all the first feature vectors in the sub-dataset is calculated as the second category feature vector of the corresponding sub-dataset, the retrieval process includes: According to the input image to be retrieved, extract features through the image-text encoder of the Clip model to obtain an image feature vector, perform similarity calculation between the obtained image feature vector and each second category vector of each first sub-data set, and obtain the top n target sub-data sets according to a preset first threshold; and The target image feature vectors are then retrieved through the index files of each target sub-dataset, and the similarity between the image feature vector and the target image feature vector corresponding to each index is calculated. According to the preset second threshold, the top m target image feature vectors in the calculation results are selected and sorted, and finally the images corresponding to the n×m target image feature vectors are obtained.

5. The design patent image retrieval method according to claim 3, characterized in that: When the fused feature vector is stored in the sub-dataset, and when the average value of all the fused feature vectors in the sub-dataset is calculated as the first category feature vector of the corresponding sub-dataset, the retrieval process includes: When an image to be retrieved and the text information corresponding to the image are input, features are extracted through the image-text encoder of the Clip model to obtain image feature vectors and text feature vectors, and a fused feature vector is calculated according to a preset weight ratio of the image feature vector and the text feature vector; the top n target sub-datasets are obtained by calculating the similarity between the fused feature vector and the first category vector of each second sub-dataset, and according to a preset third threshold, the target sub-datasets are obtained, and then the target image feature vectors are retrieved through the index files of each target sub-dataset, and the similarity between the fused feature vector and the fused feature vector corresponding to the index in each second sub-dataset is calculated, and according to the preset fourth threshold, the top m target image feature vectors in the calculation result are selected, and finally images corresponding to n×m target image feature vectors are obtained.

6. A device for searching appearance patent images, characterized in that include: A data set construction module, used to construct a data set consisting of appearance images and text label data pairs representing attributes of the appearance images; A model training module is used to input the data set into the initial Clip model to train it, so as to obtain a graphic encoder of the trained Clip model; A feature vector acquisition module is used to input the data set into the image-text encoder of the trained Clip model to generate a first feature vector corresponding to the appearance image feature and a second feature vector corresponding to the text feature, and perform a weighted sum calculation on the first feature vector and the second feature vector according to a preset vector weight ratio to obtain a fused feature vector; A feature clustering module is used to cluster the fused feature vectors based on a clustering algorithm to obtain a sub-dataset, wherein the fused feature vectors are stored in the sub-dataset, and the average value of all the fused feature vectors in the sub-dataset is calculated as the first category feature vector of the sub-dataset; or the first feature vector of the image feature is stored in the sub-dataset, and the average value of all the first feature vectors in the sub-dataset is calculated as the second category feature vector of the sub-dataset; as well as The retrieval module is used to input the image to be retrieved into the image-text encoder of the trained Clip model to obtain an image feature vector, perform similarity calculation between the image feature vector and the category vector of the sub-dataset to obtain the top n target sub-datasets, and then perform similarity calculation between the image feature vector and each feature vector stored in the target sub-dataset to obtain the top m target image feature vectors, thereby obtaining the retrieved image results, wherein n is a natural number greater than 1, and m is a natural number less than or equal to n.

7. The design patent image retrieval device according to claim 6, characterized in that: It also includes a weight setting module, which is used to set the dimensions and weights of the first eigenvector and the second eigenvector, wherein the weights of the first eigenvector and the second eigenvector are set to 0.7 and 0.3 respectively.

8. The design patent image retrieval device according to claim 6, characterized in that: The sub-dataset stores the fused feature vector or the first feature vector corresponding to the image feature, and indexes the feature vectors contained in the sub-dataset through the faiss library and stores them.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the appearance patent image retrieval method as described in any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the appearance patent image retrieval method as described in any one of claims 1 to 5 is implemented.

Images