Retrieval method and system based on semantic keyword classification and multi-language intelligent icons

Abstract

The invention discloses a retrieval method and system based on semantic keyword classification and multi-language intelligent icons, and belongs to the technical field of icon retrieval. The method comprises the following steps: constructing a CLIP model supporting multiple languages; collecting a multi-source icon data set, and taking the multi-source icon data set as a training set to train the CLIP model to obtain a trained CLIP model; processing the icons in the icon library and the associated text tags by using the trained CLIP model to generate a corresponding picture feature vector library and a text feature vector library; extracting a to-be-matched text feature vector of the input text by using the trained CLIP model, and matching the to-be-matched text feature vector with a text feature vector in a text feature vector library to obtain a preliminary matching result; and calculating the similarity between the picture feature vector in the icon set corresponding to the preliminary matching result and the to-be-matched text feature vector in the semantic space to obtain a final matching result. According to the method, the limitation of a single language is broken through, and the retrieval accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of icon retrieval, and more particularly to a retrieval method and system based on semantic keyword classification and multilingual intelligent icons. BACKGROUND

[0002] At present, some icon retrieval systems in the prior art use a CLIP (Contrastive Language-Image Pretraining) model. The CLIP model is a multi-modal model based on contrastive learning, which can map text and images to the same semantic space, thereby realizing image retrieval based on text description

[0003] However, in practical applications, there are many problems in directly using the pre-trained CLIP model. First, the training data used by the pre-trained CLIP model is a general large-scale data set, and the training target is to cover a wide range of natural language and image semantic relationships, and it is not targeted at icon resources in a specific scenario. Icon has a unique visual language and semantic expression, for example, in mobile application icons, different shapes, colors and element combinations represent specific application functions and meanings, which are quite different from general images. Secondly, the prior art does not retrain the CLIP model for icon data, making it difficult for the CLIP model to accurately understand the semantic information contained in the icon, resulting in low accuracy and relevance of the retrieval results in the icon retrieval task, and failing to meet the user's demand for efficient retrieval of icon resources. At the same time, most of the existing systems only support single language retrieval, and in a multilingual environment, they cannot provide convenient icon retrieval services for users with different language backgrounds. SUMMARY

[0004] Therefore, the present application provides a retrieval method and system based on semantic keyword classification and multilingual intelligent icons to at least solve some of the technical problems in the background art.

[0005] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] The present application first discloses a retrieval method based on semantic keyword classification and multilingual intelligent icons, comprising the following steps:

[0007] Constructing a CLIP model supporting multiple languages;

[0008] Collecting a multi-source icon data set and using it as a training set to train the CLIP model to obtain a trained CLIP model;

[0009] Using the trained CLIP model to process the icons and associated text labels in the icon library to generate corresponding picture feature vector library and text feature vector library;

[0010] extracting a to-be-matched text feature vector of the input text by using the trained CLIP model, matching the to-be-matched text feature vector with text feature vectors in the text feature vector library, and obtaining a preliminary matching result meeting a set text feature vector threshold;

[0011] calculating a similarity of the picture feature vector in the icon set corresponding to the preliminary matching result and the to-be-matched text feature vector in a semantic space, and obtaining a final icon matched with the input text.

[0012] Further, in the step of constructing the CLIP model supporting multiple languages, the constructed CLIP model comprises an image encoder, a text encoder, and a Tokenizer component supporting multiple languages, and an output of the Tokenizer component is used as an input of the text encoder.

[0013] Further, the Tokenizer component supporting multiple languages is obtained by the following steps:

[0014] Collecting original texts containing icon keywords and translating them into specified multiple different language texts, including simplified Chinese, traditional Chinese, Japanese, Korean, English, French, Italian, and Indonesian;

[0015] Using a BPE algorithm to jointly train the multiple different language texts to generate a Tokenizer component shared and supporting multiple languages.

[0016] Further, in the step of constructing the CLIP model supporting multiple languages, the text encoder adopts a BERT-base-multilingual-cased module, a number of layers of a Transformer submodule in the BERT-base-multilingual-cased module is adjusted from 12 layers to 8 layers, and a hidden layer dimension is reduced from 768 to 512.

[0017] Further, in the step of training the CLIP model by using a multi-source icon dataset as a training set, the multi-source icon dataset comprises icon resources in different application fields, styles, and functions.

[0018] Further, the step of calculating the similarity of the picture feature vector in the icon set corresponding to the preliminary matching result and the to-be-matched text feature vector in the semantic space specifically comprises:

[0019] calculating a cosine similarity of the to-be-matched text feature vector of the input text and each picture feature vector in the icon set in the semantic space;

[0020] According to the cosine similarity result, an icon retrieval result most relevant to the input text is obtained.

[0021] The application also discloses a semantic keyword classification and multilingual intelligent icon retrieval system.

[0022] A model construction module is configured to construct a CLIP model supporting multiple languages.

[0023] A data collection module is configured to collect a multi-source icon dataset as a training set of the CLIP model supporting multiple languages.

[0024] A model training module is configured to train the CLIP model supporting multiple languages by using the training set obtained by the data collection module.

[0025] A picture feature vector library is configured to store picture feature vectors generated by processing icons in an icon library by using the trained CLIP model.

[0026] A text feature vector library is configured to store text label vectors generated by processing labels corresponding to icons in the icon library by using the trained CLIP model.

[0027] A text preliminary retrieval module is configured to extract a text feature vector to be matched from an input text by using the trained CLIP model, match the text feature vector to be matched with text feature vectors in the text feature vector library, and obtain a preliminary matching result meeting a set text feature vector threshold.

[0028] A text final retrieval module is configured to calculate the similarity of picture feature vectors in an icon set corresponding to the preliminary matching result and a text feature vector to be matched in a semantic space, and obtain a final icon matched with the input text.

[0029] Preferably, the CLIP model constructed in the model construction module comprises an image encoder, a text encoder and a tokenizer component supporting multiple languages, and the output of the tokenizer component supporting multiple languages is used as the input of the text encoder.

[0030] Preferably, the tokenizer component supporting multiple languages is obtained by the following steps:

[0031] Raw texts containing icon keywords are collected and translated into specified multiple different language texts, including simplified Chinese, traditional Chinese, Japanese, Korean, English, French, Italian and Indonesian.

[0032] The multiple different language texts are jointly trained by using a BPE algorithm, and a shared tokenizer component supporting multiple languages is generated.

[0033] Preferably, the text encoder adopts a BERT-base-multilingual-cased module, the number of layers of the Transformer architecture in the BERT-base-multilingual-cased module is adjusted from 12 layers to 8 layers, and the hidden layer dimension is reduced from 768 to 512.

[0034] Compared with the prior art, the retrieval method and system based on semantic keyword classification and multilingual intelligent icons provided by the application have the following beneficial effects:

[0035] The application realizes icon retrieval in a multilingual environment by constructing a CLIP model supporting multiple languages, breaks the limitation of a single language, and greatly expands the system application scenarios and user groups.

[0036] The application adopts a two-level retrieval mechanism of text search text rough sorting and text search image fine sorting to improve the retrieval accuracy, which is of great significance in large-scale data retrieval scenarios.

[0037] The application can learn complex semantic associations by introducing a domain-specific dataset, thereby showing stronger generalization ability when facing diversified queries.

[0038] The application particularly emphasizes training using its own icon data to enhance the model's understanding ability of icon images. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only a part of the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0040] Figure 1 The retrieval method based on semantic keyword classification and multilingual intelligent icons provided by the application is shown in the whole process schematic diagram.

[0041] Figure 2 The text editor architecture provided by the application is shown in the schematic diagram. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0043] This invention discloses a retrieval method based on semantic keyword classification and multilingual intelligent icons, such as... Figure 1 As shown, it includes the following steps:

[0044] Build a CLIP model that supports multiple languages;

[0045] Collect a multi-source icon dataset and use it as the training set to train the CLIP model, thus obtaining a trained CLIP model.

[0046] The trained CLIP model is used to process the icons and associated text tags in the icon library to generate corresponding image feature vector libraries and text feature vector libraries.

[0047] The trained CLIP model is used to extract the text feature vector of the input text to be matched. The text feature vector to be matched is matched with the text feature vector in the text feature vector library to obtain a preliminary matching result that meets the set text feature vector threshold.

[0048] Calculate the similarity in semantic space between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, and obtain the final icon that matches the input text.

[0049] The steps of the present invention will be described in detail below.

[0050] This invention first constructs a CLIP model training method suitable for icon data training. In this method, the structure of the original CLIP model is specifically adjusted, replacing the original English-only tokenizer with a multilingual tokenizer supporting multiple languages, and optimizing the text encoder and image encoder to better adapt to the characteristics of icon data. Specifically, as follows... Figure 2 As shown, the text encoder uses BERT-base-multilingual-cased to support multilingual environments. The embedding dimension was adjusted from [49407, 512] to [106542, 512]. Simultaneously, the number of Transformer layers was reduced from 12 to 8, and the hidden layer dimension was decreased from 768 to 512, increasing inference speed without sacrificing performance. For the image encoder, the model structure remained largely unchanged. Based on clip-ViT-L-14, the model's structural parameters were modified to reduce parameters and improve speed. The number of Transformer layers was reduced from 24 to 16, and the hidden layer dimension was adjusted from 768 to 512. This reduces model computation and improves training and inference efficiency while maintaining image feature extraction capabilities.

[0051] Next, the application uses its own icon data to train CLIP: In this step, a large-scale icon dataset of about 3w is collected and organized, which covers icons of different application fields, styles and functions, and the icons are annotated in detail, including function keywords, style keywords, etc. The keywords are translated into 8 languages. The custom CLIP training framework is supervised and retrained using this icon dataset. Through the back propagation algorithm, the model parameters are constantly adjusted, so that the model can learn the corresponding relationship between the icon and the semantic keyword, thereby enhancing the understanding of the icon semantics. During the training process, the learning rate is set to 0.0005, the Adam optimizer is used, the training batch size is set to 32, and a total of 56 epochs are trained on an A10 machine to ensure that the model fully learns the icon data features.

[0052] It should be noted that the CLIP model disclosed in the application supports multiple languages: The text encoding module of the CLIP model is improved, including Tokenizer and text encoder. At the same time, a multi-language image-text pair dataset is introduced during training. Thus, the icon retrieval function in a multi-language environment is realized. At the same time, the semantic annotation of the icon data is also extended to multiple languages to ensure that different language texts can be accurately matched to the corresponding icons.

[0053] Tokenizer is a text encoding tool trained from a large-scale text corpus by unsupervised learning algorithm. Its core components include a vocabulary and a set of merge rules. In this project, the construction of Tokenizer is based on the text part of the image-text pair dataset, which is used to convert natural language text into numerical representation that can be processed by the model.

[0054] In order to support multi-language understanding ability, the original text containing icon keywords is first translated into eight languages including Simplified Chinese (covering: Simplified Chinese, Traditional Chinese, Japanese, Korean, English, French, Italian, Indonesian), and then the BPE (Byte-Pair Encoding) algorithm is used to jointly train the multi-language text, and finally a shared multi-language Tokenizer is generated. The icon keywords in this step include icon categories, application attributes, etc. For example, a "leaf icon" has the category keyword "nature" and the application attribute "environmental protection".

[0055] The Tokenizer has a vocabulary size of 50,000 and learns an optimal set of subword splitting and merging rules during training. Compared to monolingual Tokenizers, the multilingual Tokenizer has the following significant differences:

[0056] Wider vocabulary coverage : The vocabulary not only contains common words and word formation patterns of a single language, but also integrates subword structures of multiple languages, with stronger language adaptability and generalization ability;

[0057] More diverse merging rules : Since the training corpus contains multiple languages, the learned subword division rules can better handle cross-language word formation differences;

[0058] Unified encoding space : Different languages of text are mapped to the same semantic space, which helps the model achieve cross-language understanding and alignment.

[0059] The final output of the Tokenizer mainly includes two core files:

[0060] 1. Vocabulary : Defines all recognizable subword units and their corresponding token IDs;

[0061] 2. Merge rules : Guide how to gradually merge the original character sequence into tokens in the vocabulary.

[0062] These two components are completely determined by the training corpus and do not depend on specific model structures, so the Tokenizer itself does not have neural network parameters or architecture design, and only exists as a pre-processing module for input text.

[0063] Vocabulary example:

[0064] "you" → 100

[0065] "good" → 200

[0066] "artificial" → 5000

[0067] "smart" → 6000

[0068] "artificial intelligence" → 9000

[0069] Tokenization and merging rules example:

[0070] "person" + "work" → "artificial"

[0071] "artificial" + "intelligence" → "artificial intelligence"

[0072] "Artificial intelligence" + "intelligence" → "artificial intelligence"

[0073] Then, text search and icon rough sorting are performed: before performing text search, a text search and icon rough sorting operation is performed. A semantic keyword classification index library is constructed, and the collected icon semantic keywords are classified according to functional categories (such as social, entertainment, and tool) and style categories (such as flat, realistic, and cartoon). When the user inputs the search text, the semantic keywords related to the search text are filtered out through text clip model matching. According to the matching degree of the keyword set and the icon semantic annotation, the icons are preliminarily sorted to obtain 20 rough sorting results.

[0074] Finally, text search and icon fine sorting are performed: the icon set after rough sorting and the search text input by the user are input into the trained CLIP model, and the similarity of the text and the icon in the semantic space is calculated. When calculating the similarity, the cosine similarity calculation method is used to quantify the similarity between the text and the icon feature vectors, and the icons are accurately sorted, and finally the icon search results most related to the search text are output. In the present application, each icon in the icon library contains corresponding labels, such as a "leaf icon", which may have labels such as "spring" or "summer", "indoor" or "outdoor", "environmental protection", and different kinds or keywords. The text encoder in the trained CLIP model processes the label text of the icon to generate a text feature vector and store it in the text feature vector library. At the same time, the image encoder in the trained CLIP model processes the image of the icon to generate a picture feature vector and store it in the picture feature vector library. In order to establish the association between the text feature vector and the picture feature vector of the same icon, it is convenient to find the corresponding icon and the picture feature vector of the icon through the text feature vector of the icon. In a specific embodiment, a unique id can be set for the same icon, and the id is attached to the text feature vector and the picture feature vector when the text feature vector and the picture feature vector are generated, so as to realize the correspondence and search of the icon, the text feature vector and the picture feature vector through the unique id.

[0075] Based on the above method, the present application also discloses a multi-language intelligent icon retrieval system based on semantic keyword classification, comprising:

[0076] A model construction module is used to construct a CLIP model supporting multiple languages;

[0077] A data acquisition module is used to acquire a multi-source icon data set as a training set of the CLIP model supporting multiple languages;

[0078] The model training module is configured to train the CLIP model supporting multiple languages by using the training set obtained by the data collection module.

[0079] The picture feature vector library is configured to store picture feature vectors generated by processing the icons in the icon library by using the trained CLIP model.

[0080] The text feature vector library is configured to store text feature vectors generated by processing the labels corresponding to the icons in the icon library by using the trained CLIP model.

[0081] The text preliminary retrieval module is configured to extract a to-be-matched text feature vector of the input text by using the trained CLIP model, match the to-be-matched text feature vector with the text feature vectors in the text feature vector library, and obtain a preliminary matching result that satisfies a set text feature vector threshold.

[0082] The text final retrieval module is configured to calculate the similarity of the picture feature vectors in the icon set corresponding to the preliminary matching result and the to-be-matched text feature vector in the semantic space, and obtain a final icon matched with the input text.

[0083] The data processing process of each module can be referred to the description in the foregoing method, which will not be repeated here.

[0084] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, the description is relatively simple because it corresponds to the method disclosed in the embodiments. The relevant parts can be referred to the description of the method.

[0085] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for semantic keyword classification and multilingual intelligent icon retrieval, characterized in that, Includes the following steps: Build a CLIP model that supports multiple languages; Collect a multi-source icon dataset and use it as the training set to train the CLIP model, thus obtaining a trained CLIP model. The trained CLIP model is used to process the icons and associated text tags in the icon library to generate corresponding image feature vector libraries and text feature vector libraries. The trained CLIP model is used to extract the text feature vector of the input text to be matched. The text feature vector to be matched is matched with the text feature vector in the text feature vector library to obtain a preliminary matching result that meets the set text feature vector threshold. Calculate the similarity in semantic space between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, and obtain the final icon that matches the input text.

2. The method for semantic keyword classification and multilingual intelligent icon retrieval according to claim 1, characterized in that, In the step of building a CLIP model that supports multiple languages, the CLIP model includes an image encoder, a text encoder, and a tokenizer component that supports multiple languages. The output of the tokenizer component is used as the input of the text encoder.

3. The method for semantic keyword classification and multilingual intelligent icon retrieval according to claim 2, characterized in that, A multilingual Tokenizer component can be obtained through the following steps: Collect raw text containing icon keywords and translate it into multiple specified languages, including Simplified Chinese, Traditional Chinese, Japanese, Korean, English, French, Italian, and Indonesian. By using the BPE algorithm to jointly train texts in multiple different languages, a shared tokenizer component that supports multiple languages ​​is generated.

4. The method for semantic keyword classification and multilingual intelligent icon retrieval according to claim 2, characterized in that, In the step of building a CLIP model that supports multiple languages, the text encoder adopts the BERT-base-multilingual-cased module. The number of Transformer sub-modules in the BERT-base-multilingual-cased module is adjusted from 12 layers to 8 layers, and the hidden layer dimension is reduced from 768 to 512.

5. The method for semantic keyword classification and multilingual intelligent icon retrieval according to claim 1, characterized in that, In the step of collecting a multi-source icon dataset and using it as a training set to train the CLIP model, the multi-source icon dataset includes icon resources from different application areas, styles, and functions.

6. The method for semantic keyword classification and multilingual intelligent icon retrieval according to claim 1, characterized in that, Calculate the semantic similarity between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, specifically including: Calculate the cosine similarity in semantic space between the feature vector of the input text to be matched and the feature vector of each image in the icon set; Based on the cosine similarity results, the icon retrieval results most relevant to the input text are obtained.

7. A semantic keyword classification and multilingual intelligent icon retrieval system, characterized in that, include: The model building module is used to build CLIP models that support multiple languages; The data acquisition module is used to collect multi-source icon datasets and use them as training sets for CLIP models that support multiple languages. The model training module is used to train a CLIP model that supports multiple languages ​​by using the training set obtained from the data acquisition module. The image feature vector library is used to store image feature vectors generated by processing icons in the icon library using a trained CLIP model. The text feature vector library is used to store text label vectors generated by processing the labels corresponding to icons in the icon library using a trained CLIP model. The preliminary text retrieval module is used to extract the text feature vector to be matched from the input text using the trained CLIP model, and to match the text feature vector to be matched with the text feature vector in the text feature vector library to obtain a preliminary matching result that meets the set text feature vector threshold. The final text retrieval module is used to calculate the similarity in semantic space between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, so as to obtain the final icon that matches the input text.

8. The semantic keyword classification and multilingual intelligent icon retrieval system according to claim 7, characterized in that, In the model building module, the CLIP model includes an image encoder, a text encoder, and a tokenizer component that supports multiple languages. The output of the tokenizer component serves as the input of the text encoder.

9. A semantic keyword classification and multilingual intelligent icon retrieval system according to claim 8, characterized in that, A multilingual Tokenizer component can be obtained through the following steps: Collect raw text containing icon keywords and translate it into multiple specified languages, including Simplified Chinese, Traditional Chinese, Japanese, Korean, English, French, Italian, and Indonesian. By using the BPE algorithm to jointly train texts in multiple different languages, a shared tokenizer component that supports multiple languages ​​is generated.

10. A semantic keyword classification and multilingual intelligent icon retrieval system according to claim 8, characterized in that, The text encoder uses the BERT-base-multilingual-cased module. The number of Transformer sub-modules in the BERT-base-multilingual-cased module has been adjusted from 12 to 8, and the hidden layer dimension has been reduced from 768 to 512.

Images