Retrieval method and system based on semantic keyword classification and multi-language intelligent icon

Abstract

The application discloses a retrieval method and system based on semantic keyword classification and multilingual intelligent icons, and belongs to the technical field of icon retrieval. The method comprises the following steps: constructing a CLIP model supporting multiple languages; collecting a multi-source icon dataset, and training the CLIP model by taking the multi-source icon dataset as a training set to obtain a trained CLIP model; processing icons in an icon library and associated text labels by using the trained CLIP model to generate a corresponding picture feature vector library and a text feature vector library; extracting a to-be-matched text feature vector of input text by using the trained CLIP model, and matching the to-be-matched text feature vector with text feature vectors in the text feature vector library to obtain a preliminary matching result; and calculating the similarity of picture feature vectors in an icon set corresponding to the preliminary matching result and the to-be-matched text feature vector in a semantic space to obtain a final matching result. The application breaks the limitation of a single language and improves the retrieval accuracy.

Description

Technical Field

[0001] This invention relates to the field of icon retrieval technology, and more specifically to a retrieval method and system based on semantic keyword classification and multilingual intelligent icons. Background Technology

[0002] Currently, some icon retrieval systems employ the CLIP (Contrastive Language-Image Pretraining) model. The CLIP model is a multimodal model based on contrastive learning that maps text and images to the same semantic space, thereby enabling image retrieval based on text descriptions.

[0003] However, directly using pre-trained CLIP models in practical applications presents several problems. First, the training data used in pre-trained CLIP models is general, large-scale datasets. Their training objective is to cover a broad range of natural language and image semantic relationships, not icon resources specific to particular scenarios. Icons, on the other hand, possess unique visual language and semantic expression. For example, in mobile application icons, different shapes, colors, and element combinations represent specific application functions and meanings, differing significantly from general images. Second, existing technologies do not retrain CLIP models specifically for icon data, making it difficult for them to accurately understand the semantic information contained in icons. This results in low accuracy and relevance of search results in icon retrieval tasks, failing to meet users' needs for efficient icon resource retrieval. Furthermore, most existing systems only support single-language retrieval, failing to provide convenient icon retrieval services for users with different language backgrounds in multilingual environments. Summary of the Invention

[0004] In view of this, the present invention provides a retrieval method and system based on semantic keyword classification and multilingual intelligent icons, which is used to at least solve some of the technical problems in the background art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention first discloses a retrieval method based on semantic keyword classification and multilingual intelligent icons, comprising the following steps:

[0007] Build a CLIP model that supports multiple languages;

[0008] Collect a multi-source icon dataset and use it as the training set to train the CLIP model, thus obtaining a trained CLIP model.

[0009] The trained CLIP model is used to process the icons and associated text tags in the icon library to generate corresponding image feature vector libraries and text feature vector libraries.

[0010] The trained CLIP model is used to extract the text feature vector of the input text to be matched. The text feature vector to be matched is matched with the text feature vector in the text feature vector library to obtain a preliminary matching result that meets the set text feature vector threshold.

[0011] Calculate the similarity in semantic space between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, and obtain the final icon that matches the input text.

[0012] Furthermore, in the step of constructing a CLIP model that supports multiple languages, the constructed CLIP model includes an image encoder, a text encoder, and a tokenizer component that supports multiple languages, with the output of the tokenizer component serving as the input of the text encoder.

[0013] Furthermore, a multilingual Tokenizer component is obtained through the following steps:

[0014] Collect raw text containing icon keywords and translate it into multiple specified languages, including Simplified Chinese, Traditional Chinese, Japanese, Korean, English, French, Italian, and Indonesian.

[0015] By using the BPE algorithm to jointly train texts in multiple different languages, a shared tokenizer component that supports multiple languages ​​is generated.

[0016] Furthermore, in the step of constructing a CLIP model that supports multiple languages, the text encoder adopts the BERT-base-multilingual-cased module, and the number of Transformer sub-modules in the BERT-base-multilingual-cased module is adjusted from 12 layers to 8 layers, and the hidden layer dimension is reduced from 768 to 512.

[0017] Furthermore, in the step of collecting a multi-source icon dataset and using it as a training set to train the CLIP model, the multi-source icon dataset includes icon resources from different application areas, styles, and functions.

[0018] Furthermore, the similarity in semantic space between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched includes:

[0019] Calculate the cosine similarity in semantic space between the feature vector of the input text to be matched and the feature vector of each image in the icon set;

[0020] Based on the cosine similarity results, the icon retrieval results most relevant to the input text are obtained.

[0021] Another aspect of this invention discloses a semantic keyword classification and multilingual intelligent icon retrieval system, comprising:

[0022] The model building module is used to build CLIP models that support multiple languages;

[0023] The data acquisition module is used to collect multi-source icon datasets and use them as training sets for CLIP models that support multiple languages.

[0024] The model training module is used to train a CLIP model that supports multiple languages ​​by using the training set obtained from the data acquisition module.

[0025] The image feature vector library is used to store image feature vectors generated by processing icons in the icon library using a trained CLIP model.

[0026] The text feature vector library is used to store text label vectors generated by processing the labels corresponding to icons in the icon library using a trained CLIP model.

[0027] The preliminary text retrieval module is used to extract the text feature vector to be matched from the input text using the trained CLIP model, and to match the text feature vector to be matched with the text feature vector in the text feature vector library to obtain a preliminary matching result that meets the set text feature vector threshold.

[0028] The final text retrieval module is used to calculate the similarity in semantic space between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, so as to obtain the final icon that matches the input text.

[0029] Preferably, in the model building module, the constructed CLIP model includes an image encoder, a text encoder, and a multilingual tokenizer component, with the output of the tokenizer component serving as the input of the text encoder.

[0030] Preferably, a multilingual Tokenizer component is obtained through the following steps:

[0031] Collect raw text containing icon keywords and translate it into multiple specified languages, including Simplified Chinese, Traditional Chinese, Japanese, Korean, English, French, Italian, and Indonesian.

[0032] The BPE algorithm is used to jointly train texts in multiple different languages, thereby generating a shared tokenizer component that supports multiple languages.

[0033] Preferably, the text encoder uses the BERT-base-multilingual-cased module, in which the number of Transformer architecture layers is adjusted from 12 to 8, and the hidden layer dimension is reduced from 768 to 512.

[0034] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a retrieval method and system based on semantic keyword classification and multilingual intelligent icons, which has the following beneficial effects:

[0035] This invention enables icon retrieval in a multilingual environment by constructing a CLIP model that supports multiple languages, breaking the limitations of a single language and greatly expanding the system's application scenarios and user base.

[0036] This invention employs a two-level retrieval mechanism of text search with coarse text ranking and text search with fine image ranking, which improves retrieval accuracy and is of great significance in large-scale data retrieval scenarios.

[0037] This invention, by introducing a domain-specific dataset, is able to learn complex semantic relationships, thereby exhibiting stronger generalization ability when faced with diverse queries.

[0038] This invention places particular emphasis on using its own icon data for training to enhance the model's ability to understand icon-type images. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the overall process of the retrieval method based on semantic keyword classification and multilingual intelligent icons provided by the present invention.

[0041] Figure 2 This is a schematic diagram of the text editor architecture provided by the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] This invention discloses a retrieval method based on semantic keyword classification and multilingual intelligent icons, such as... Figure 1 As shown, it includes the following steps:

[0044] Build a CLIP model that supports multiple languages;

[0045] Collect a multi-source icon dataset and use it as the training set to train the CLIP model, thus obtaining a trained CLIP model.

[0046] The trained CLIP model is used to process the icons and associated text tags in the icon library to generate corresponding image feature vector libraries and text feature vector libraries.

[0047] The trained CLIP model is used to extract the text feature vector of the input text to be matched. The text feature vector to be matched is matched with the text feature vector in the text feature vector library to obtain a preliminary matching result that meets the set text feature vector threshold.

[0048] Calculate the similarity in semantic space between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, and obtain the final icon that matches the input text.

[0049] The steps of the present invention will be described in detail below.

[0050] This invention first constructs a CLIP model training method suitable for icon data training. In this method, the structure of the original CLIP model is specifically adjusted, replacing the original English-only tokenizer with a multilingual tokenizer supporting multiple languages, and optimizing the text encoder and image encoder to better adapt to the characteristics of icon data. Specifically, as follows... Figure 2 As shown, the text encoder uses BERT-base-multilingual-cased to support multilingual environments. The embedding dimension was adjusted from [49407, 512] to [106542, 512]. Simultaneously, the number of Transformer layers was reduced from 12 to 8, and the hidden layer dimension was decreased from 768 to 512, increasing inference speed without sacrificing performance. For the image encoder, the model structure remained largely unchanged. Based on clip-ViT-L-14, the model's structural parameters were modified to reduce parameters and improve speed. The number of Transformer layers was reduced from 24 to 16, and the hidden layer dimension was adjusted from 768 to 512. This reduces model computation and improves training and inference efficiency while maintaining image feature extraction capabilities.

[0051] Next, this invention trains CLIP using its own icon data: In this step, a large-scale icon dataset of approximately 30,000 icons was collected and organized. This dataset covers icon resources from different application areas, styles, and functions, and the icons are semantically annotated in detail, including functional keywords, style keywords, etc. The keywords were translated into eight languages. The custom CLIP training framework is then retrained using this icon dataset in a supervised manner. The model parameters are continuously adjusted through backpropagation, enabling the model to learn the correspondence between icons and semantic keywords, thereby enhancing its ability to understand icon semantics. During training, a learning rate of 0.0005 was set, the Adam optimizer was used, the training batch size was set to 32, and a total of 56 epochs were trained on an A10 machine to ensure that the model fully learns the features of the icon data.

[0052] It should be noted that the CLIP model disclosed in this invention supports multiple languages ​​and improves the text encoding module of the CLIP model, including a tokenizer and a text encoder. Furthermore, a multilingual image-text pair dataset is introduced during training, thereby enabling icon retrieval in a multilingual environment. The semantic annotation of the icon data has also been extended to multiple languages ​​to ensure that text in different languages ​​can accurately match the corresponding icons.

[0053] Tokenizer is a text encoding tool trained from large-scale text corpora using unsupervised learning algorithms. Its core components include a vocabulary and a set of merge rules. In this project, Tokenizer is built based on the text portion of an image-text pair dataset to convert natural language text into a numerical representation that the model can process.

[0054] To support multilingual understanding, the original text containing icon keywords is first translated into eight languages, including Simplified Chinese, Traditional Chinese, Japanese, Korean, English, French, Italian, and Indonesian. Then, the Byte-Pair Encoding (BPE) algorithm is used to jointly train the multilingual text, ultimately generating a shared multilingual tokenizer. Icon keywords in this step include icon type and application attributes. For example, a "leaf icon" might have the category keyword "nature" and the application attribute "environmental protection."

[0055] The vocabulary size of this Tokenizer is set to 50,000, and a set of optimal sub-word splitting and merging rules are learned during training. Compared with the monolingual Tokenizer, the multilingual Tokenizer has the following significant differences:

[0056] Wider vocabulary coverage: The vocabulary not only contains common words and word formation methods of a single language, but also integrates sub-word structures of multiple languages, with stronger language adaptability and generalization ability;

[0057] More diverse merging rules: Since the training corpus contains multiple languages, the learned sub-word partitioning rules can better handle cross-language word formation differences;

[0058] Unified encoding space: Texts in different languages are mapped to the same semantic space, which helps the model achieve cross-language understanding and alignment.

[0059] The final output of the Tokenizer mainly includes two core files:

[0060] 1. Vocabulary: Defines all recognizable sub-word units and their corresponding token IDs;

[0061] 2. Merge rules: Guides how to gradually merge the original character sequence into tokens in the vocabulary.

[0062] These two components are completely determined by the training corpus and do not depend on a specific model structure. Therefore, the Tokenizer itself does not have neural network parameters or architecture design and only exists as a preprocessing module for input text.

[0063] Example of vocabulary:<00, "you" → 100

[0065] "good" → 200

[0066] "artificial" → 5000

[0067] "intelligent" → 6000

[0068] "artificial intelligence" → 9000

[0069] Example of word segmentation and merging rules:

[0070] "person" + "work" → "artificial"

[0071] "artificial" + "intelligent" → "artificial intelligent"

[0072] ​​"Artificial Intelligence" + "Ability" → "Artificial Intelligence"

[0073] Then, a preliminary text search ranking is performed: Before ranking the icons based on text search results, a preliminary text search ranking is conducted. A semantic keyword classification index is built, categorizing the collected icon semantic keywords according to functional categories (such as social, entertainment, tools, etc.) and style categories (such as flat, realistic, cartoon, etc.). When a user enters search text, a text clip model is used for matching to filter out a set of keywords semantically related to the search text. Based on the degree of matching between the keyword set and the icon semantic annotations, the icons are initially ranked, resulting in 20 preliminary ranking results.

[0074] Finally, text-based image ranking is performed: the coarsely ranked icon set and the user-input search text are input into the trained CLIP model. The similarity between text and icons in the semantic space is calculated using cosine similarity to quantify the similarity between text and icon feature vectors, thus precisely ranking the icons and ultimately outputting the most relevant icon search results to the search text. In this invention, each icon in the icon library contains a corresponding tag. For example, a "leaf icon" might have tags such as "spring" or "summer," "indoor" or "outdoor," or "environmental protection," among other different categories or keywords. The text encoder in the trained CLIP model processes the icon's tag text, generating text feature vectors which are stored in the text feature vector library. Simultaneously, the image encoder in the trained CLIP model processes the icon's image, generating image feature vectors which are stored in the image feature vector library. To establish a correlation between the text feature vector and the image feature vector of the same icon, and to facilitate finding the corresponding icon and its image feature vector through the text feature vector of an icon, in one specific implementation, a unique ID can be set for the same icon. When generating the text feature vector and the image feature vector, the ID is attached to the text feature vector and the image feature vector. The unique ID enables the correspondence and lookup of the icon, text feature vector, and image feature vector.

[0075] Based on the above method, the present invention also discloses a semantic keyword classification and multilingual intelligent icon retrieval system, comprising:

[0076] The model building module is used to build CLIP models that support multiple languages;

[0077] The data acquisition module is used to collect multi-source icon datasets and use them as training sets for CLIP models that support multiple languages.

[0078] The model training module is used to train a CLIP model that supports multiple languages ​​by using the training set obtained from the data acquisition module.

[0079] The image feature vector library is used to store image feature vectors generated by processing icons in the icon library using a trained CLIP model.

[0080] The text feature vector library is used to store text label vectors generated by processing the labels corresponding to icons in the icon library using a trained CLIP model.

[0081] The preliminary text retrieval module is used to extract the text feature vector to be matched from the input text using the trained CLIP model, and to match the text feature vector to be matched with the text feature vector in the text feature vector library to obtain a preliminary matching result that meets the set text feature vector threshold.

[0082] The final text retrieval module is used to calculate the similarity in semantic space between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, so as to obtain the final icon that matches the input text.

[0083] For details on the data processing procedures of each module, please refer to the description in the previous section on methods; it will not be repeated here.

[0084] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0085] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for semantic keyword classification and multilingual intelligent icon retrieval, characterized in that, Includes the following steps: A CLIP model supporting multiple languages ​​is constructed. The CLIP model includes an image encoder, a text encoder, and a tokenizer component supporting multiple languages. The output of the tokenizer component serves as the input of the text encoder. The text encoder adopts the BERT-base-multilingual-cased module. The number of layers in the Transformer submodule of the BERT-base-multilingual-cased module is adjusted from 12 to 8, and the hidden layer dimension is reduced from 768 to 512. The number of layers in the Transformer of the image encoder is adjusted from 24 to 16, and the hidden layer dimension is adjusted from 768 to 512. Collect a multi-source icon dataset and use it as the training set to train the CLIP model, thus obtaining a trained CLIP model. The trained CLIP model is used to process the icons and associated text tags in the icon library to generate corresponding image feature vector libraries and text feature vector libraries. The trained CLIP model is used to extract the text feature vector of the input text to be matched. The text feature vector to be matched is matched with the text feature vector in the text feature vector library to obtain a preliminary matching result that meets the set text feature vector threshold. Calculate the similarity in semantic space between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, and obtain the final icon that matches the input text.

2. The method for semantic keyword classification and multilingual intelligent icon retrieval according to claim 1, characterized in that, A multilingual Tokenizer component can be obtained through the following steps: Collect raw text containing icon keywords and translate it into multiple specified languages, including Simplified Chinese, Traditional Chinese, Japanese, Korean, English, French, Italian, and Indonesian. By using the BPE algorithm to jointly train texts in multiple different languages, a shared tokenizer component that supports multiple languages ​​is generated.

3. The method for semantic keyword classification and multilingual intelligent icon retrieval according to claim 1, characterized in that, In the step of collecting a multi-source icon dataset and using it as a training set to train the CLIP model, the multi-source icon dataset includes icon resources from different application areas, styles, and functions.

4. The method for semantic keyword classification and multilingual intelligent icon retrieval according to claim 1, characterized in that, Calculate the semantic similarity between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, specifically including: Calculate the cosine similarity in semantic space between the feature vector of the input text to be matched and the feature vector of each image in the icon set; Based on the cosine similarity results, the icon retrieval results most relevant to the input text are obtained.

5. A semantic keyword classification and multilingual intelligent icon retrieval system, characterized in that, include: The model building module is used to build CLIP models that support multiple languages. The CLIP model includes an image encoder, a text encoder, and a tokenizer component that supports multiple languages. The output of the tokenizer component serves as the input of the text encoder. The text encoder uses the BERT-base-multilingual-cased module. The number of layers in the Transformer submodule of the BERT-base-multilingual-cased module has been adjusted from 12 to 8, and the hidden layer dimension has been reduced from 768 to 512. The number of layers in the Transformer of the image encoder has been adjusted from 24 to 16, and the hidden layer dimension has been adjusted from 768 to 512. The data acquisition module is used to collect multi-source icon datasets and use them as training sets for CLIP models that support multiple languages. The model training module is used to train a CLIP model that supports multiple languages ​​by using the training set obtained from the data acquisition module. The image feature vector library is used to store image feature vectors generated by processing icons in the icon library using a trained CLIP model. The text feature vector library is used to store text label vectors generated by processing the labels corresponding to icons in the icon library using a trained CLIP model. The preliminary text retrieval module is used to extract the text feature vector to be matched from the input text using the trained CLIP model, and to match the text feature vector to be matched with the text feature vector in the text feature vector library to obtain a preliminary matching result that meets the set text feature vector threshold. The final text retrieval module is used to calculate the similarity in semantic space between the image feature vectors in the icon set corresponding to the preliminary matching results and the feature vectors of the text to be matched, so as to obtain the final icon that matches the input text.

6. The semantic keyword classification and multilingual intelligent icon retrieval system according to claim 5, characterized in that, A multilingual Tokenizer component can be obtained through the following steps: Collect raw text containing icon keywords and translate it into multiple specified languages, including Simplified Chinese, Traditional Chinese, Japanese, Korean, English, French, Italian, and Indonesian. By using the BPE algorithm to jointly train texts in multiple different languages, a shared tokenizer component that supports multiple languages ​​is generated.

Images