Trademark retrieval library generation method and device based on multi-modal representation generation model

Abstract

The application discloses a trademark retrieval library generation method and device based on a multi-modal feature generation model, constructs a trademark image-text pair according to the Vienna graphic element classification standard, and trains a multi-modal feature generation model; an initial trademark retrieval library is constructed by using the trained multi-modal feature generation model; trademark rejection data is obtained, and the trained multi-modal feature generation model is fine-tuned by using an adapter technology and a triple comparison learning method, so that an optimized multi-modal feature generation model is obtained; the optimized multi-modal feature generation model is used to regenerate a trademark image feature vector, and the vector is input into a vector database to obtain a final trademark retrieval library. The multi-modal feature generation model is fine-tuned twice by using the trademark image-text pair and the trademark rejection data, so that the overall retrieval effect of the trademark retrieval library constructed by using the multi-modal feature generation model is improved, and thus the trademark retrieval result is more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of trademark retrieval, and particularly relates to a trademark retrieval library generation method and device based on a multi-modal representation generation model. BACKGROUND

[0002] In the field of trademarks, picture similarity comparison mostly relies on manual operation. Specifically, when a new trademark is submitted, it needs to be compared with a large number of registered trademark pictures for similarity. The existing technology usually requires manual separation of graphic elements and text elements in the submitted trademark picture, and then comparison with registered trademark pictures and texts respectively. This process requires a large amount of manual operation, and the accuracy of the comparison result often depends on the experience and carefulness of the comparison staff.

[0003] Although some existing technologies begin to use multi-modal models (combine images and texts for similarity detection), these methods generally only make simple fine-tuning on trademark pictures, and lack in-depth analysis of the characteristics of trademark pictures. Traditional deep learning methods (such as CNN, MAE, MoCo, etc.) are also used as auxiliary tools in some existing technologies for similarity comparison of trademark pictures, but the effect is often not ideal. The reasons are as follows:

[0004] 1. Single-modal image encoding model: for example, ResNet, MAE, MoCo, etc. image encoders based on CNN, which cannot understand the specific graphic and text elements contained in the trademark image. These models pay more attention to the overall structure of the image, but lack semantic understanding, so their performance in trademark retrieval is not good.

[0005] 2. Multi-modal model: for example, CLIP (combining a ViT-based image encoder and a BERT text encoder), which has made some progress in aligning the image-text feature space, but still lacks deep fusion. In particular, the limitation of the BERT text understanding ability makes the model unable to effectively extract the content in the image when facing abstract and complex images. Therefore, the effect of models such as CLIP in trademark retrieval is also limited.

[0006] 3. Artificial rules for trademark retrieval: trademark similarity determination is often affected by human rules such as the Trademark Examination and Adjudication Guidelines, and these rules are difficult to effectively embed into the model. This leads to a possible deviation between the model's retrieval results and the expected results of manual work. For example, a trademark picture may contain multiple graphic elements, but some graphic elements may not be important and need not be specially focused on during retrieval. If these unimportant elements are integrated into the feature vector, it may cause deviation in the retrieval results. SUMMARY

[0007] To this end, the application provides a trademark retrieval library generation method and device based on a multi-modal representation generation model to solve the problem of inaccurate trademark retrieval results in the prior art.

[0008] To achieve the above object, the application provides the following technical solution:

[0009] In a first aspect, a trademark retrieval library generation method based on a multi-modal representation generation model comprises:

[0010] Step 1: Obtain a trademark image and label and classify the trademark image according to the Vienna graphic element classification standard to obtain a trademark image-text pair; the trademark image includes a graphic trademark image, a text trademark image, and a combined trademark image;

[0011] Step 2: Train a multi-modal representation generation model according to the trademark image-text pair to obtain a trained multi-modal representation generation model; the multi-modal representation generation model includes a visual coding module and a cross-modal generation module; the training process specifically includes: extracting a high-dimensional visual feature vector of the trademark image using the visual coding module, and converting the guiding text instructions into a text vector sequence through a tokenizer; after dimension alignment through a linear projection layer, the high-dimensional visual feature vector and the text vector sequence are spliced and fused according to the trademark image-text pair to obtain a joint feature; the joint feature is input into the cross-modal generation module to generate a complete description of the graphic elements and text elements corresponding to the trademark image;

[0012] Step 3: Generate a trademark image feature vector using the trained multi-modal representation generation model and record it in a vector database to obtain an initial trademark retrieval library;

[0013] Step 4: Obtain trademark rejection data, select a trademark similar to the trademark rejection data from the initial trademark retrieval library as a negative sample, and select a trademark corresponding to the trademark rejection data in the rejection data as a positive sample, and fine-tune the trained multi-modal representation generation model using adapter technology and a triple comparison learning method to obtain an optimized multi-modal representation generation model;

[0014] Step 5: Re-generate a trademark image feature vector using the optimized multi-modal representation generation model and record it in a vector database to obtain a final trademark retrieval library.

[0015] As a preferred, in step 2, the visual coding module adopts a CLIP pre-trained Vision Transformer model.

[0016] As a preferred, in step 2, the cross-modal generation module adopts a pre-trained large language model based on a decoder architecture.

[0017] As preferred, in step 2, when training the multi-modal representation generation model according to the trademark image-text pair, the loss function is:

[0018]

[0019] wherein,

[0020]

[0021]

[0022] 、 represents a hyperparameter, L ce represents a difference between the model prediction distribution and the true distribution, N represents the batch size, C represents the size of the vocabulary, y ic represents the true label of the i-th sample belonging to the category C, represents the probability predicted by the model, represents the contrastive learning loss, represents the positive sample corresponding to x, represents the negative sample corresponding to x.

[0023] As preferred, in step 3, the vector database adopts a milvus vector database.

[0024] As preferred, step 4 specifically includes:

[0025] Step 401: obtaining trademark rejection data, selecting trademarks similar to the trademark rejection data from the initial trademark retrieval library as negative samples, and selecting trademarks corresponding to the trademark rejection data in the rejection data as positive samples;

[0026] Step 402: inputting the trademark rejection data, the positive sample and the negative sample into the trained multi-modal representation generation model respectively to obtain a plurality of multi-modal representations;

[0027] Step 403: inputting the plurality of multi-modal representations into the adapter module respectively to obtain a positive sample feature vector, a rejected trademark feature vector and a negative sample feature vector;

[0028] Step 404: using a triplet contrastive learning method to perform contrastive learning on the positive sample feature vector, the rejected trademark feature vector and the negative sample feature vector, and updating the adapter module parameters.

[0029] As preferred, in step 404, when performing contrastive learning, the loss function is:

[0030]

[0031] wherein A represents a positive sample feature vector, P represents a rejected trademark feature vector, N represents a negative sample feature vector, respectively represent the distance between the anchor sample and the positive sample, the distance between the anchor sample and the negative sample in the embedding space, and margin represents a preset threshold value for controlling the difference between the positive sample and the negative sample.

[0032] As a preference, the adapter module consists of two feedforward sub-layers.

[0033] In a second aspect, a trademark retrieval library generation device based on a multi-modal representation generation model comprises:

[0034] A trademark image-text pair construction module is configured to obtain a trademark image, and label and classify the trademark image according to the Vienna graphic element classification standard to obtain a trademark image-text pair. The trademark image includes a graphic trademark image, a text trademark image, and a combined trademark image.

[0035] A multi-modal representation generation model training module is configured to train a multi-modal representation generation model according to the trademark image-text pair to obtain a trained multi-modal representation generation model. The multi-modal representation generation model includes a visual coding module and a cross-modal generation module. The training process specifically includes: extracting a high-dimensional visual feature vector of the trademark image using the visual coding module, and converting a guiding text instruction into a text vector sequence through a word segmenter; performing dimension alignment on the high-dimensional visual feature vector and the text vector sequence through a linear projection layer after the trademark image-text pair, and then performing splicing and fusion to obtain a joint feature; inputting the joint feature into the cross-modal generation module to generate a complete description of the graphic elements and text elements corresponding to the trademark image.

[0036] An initial trademark retrieval library construction module is configured to generate a trademark image feature vector using the trained multi-modal representation generation model, and record the vector in a vector database to obtain an initial trademark retrieval library.

[0037] A multi-modal representation generation model optimization module is configured to obtain trademark rejection data, select a trademark similar to the trademark rejection data from the initial trademark retrieval library as a negative sample, select a trademark corresponding to the trademark rejection data in the rejection data as a positive sample, and fine-tune the trained multi-modal representation generation model using an adapter technology and a triplet contrast learning method to obtain an optimized multi-modal representation generation model.

[0038] A trademark retrieval library optimization module is configured to regenerate a trademark image feature vector using the optimized multi-modal representation generation model, and record the vector in a vector database to obtain a final trademark retrieval library.

[0039] ​In a third aspect, a computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of a method for generating a trademark search library based on a multimodal representation generation model.

[0040] Compared with the prior art, this application has at least the following beneficial effects:

[0041] The present application provides a trademark search library generation method and device based on a multimodal representation generation model. The method comprises the following steps: obtaining trademark images and annotating and classifying them according to the Vienna Classification Standard for Graphic Elements to obtain trademark image-text pairs; training a multimodal representation generation model based on the trademark image-text pairs; generating trademark image feature vectors using the trained multimodal representation generation model and entering them into a vector database to obtain an initial trademark search library; obtaining trademark rejection data, selecting trademarks similar to the trademark rejection data from the initial trademark search library as negative samples, and selecting trademarks corresponding to the trademark rejection data from the rejection data as positive samples, and fine-tuning the trained multimodal representation generation model using adapter technology and triplet contrast learning methods to obtain an optimized multimodal representation generation model; and regenerating trademark image feature vectors using the optimized multimodal representation generation model and entering them into a vector database to obtain a final trademark search library. The present application uses trademark image-text pairs and trademark rejection data to perform two fine-tuning operations on the multimodal representation generation model, thereby improving the overall search effect of the trademark search library constructed using the multimodal representation generation model, thereby making the trademark search results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] To more intuitively illustrate the prior art and the present application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be considered as limiting conditions for implementing the present application. For example, based on the technical concepts disclosed in this application and the exemplary drawings, those skilled in the art are capable of easily making routine adjustments or further optimizations to the addition / reduction / attribution division of certain units (components), the specific shapes, positional relationships, connection methods, and dimensional ratios.

[0043] Figure 1 A basic flow chart of a trademark search library generation method based on a multimodal representation generation model provided in Example 1 of the present application;

[0044] Figure 2 A detailed flow chart of a method for generating a trademark search library based on a multimodal representation generation model provided in Example 1 of the present application;

[0045] Figure 3 A flowchart of using trademark image-text pairs to train a multimodal representation generation model provided in Example 1 of the present application;

[0046] Figure 4 The flowchart for fine-tuning a multi-modal representation generation model using trademark rejection data provided by Embodiment One of the present application. DETAILED DESCRIPTION

[0047] The present application will be further described in detail below with reference to specific embodiments in conjunction with the accompanying drawings.

[0048] In the description of the present application: unless otherwise specified, the meaning of "multiple" is two or more. The terms "first", "second", "third", etc. in the present application are intended to distinguish the objects referred to, and do not have a special meaning in the technical connotation aspect (for example, it should not be understood as an emphasis on importance or order, etc.). The expressions "include", "contain", "have" and the like also mean "not limited to" (certain units, components, materials, steps, etc.).

[0049] The terms such as "upper", "lower", "left", "right", "intermediate" and the like referred to in the present application are generally indications of the relative positional relationship for the purpose of intuitive understanding with reference to the drawings, and are not absolute limitations on the positional relationship in the actual product.

[0050] Embodiment One

[0051] Please refer to Figure 1 and Figure 2 The present embodiment provides a trademark retrieval library generation method based on a multi-modal representation generation model, which provides a systematic solution to the problems of cross-modal semantic matching, feature space alignment and trademark rejection data adaptability optimization in the trademark examination scene, and is suitable for trademark examination, brand protection and other scenes, including:

[0052] S1: Obtain a trademark image, and label and classify the trademark image according to the Vienna graphic element classification standard to obtain a trademark image-text pair; the trademark image includes a graphic trademark image, a text trademark image and a combined trademark image;

[0053] Specifically, this step constructs a trademark image-text pair training set by collecting trademark images, and the collected trademark images need to cover graphic trademark images (geometric figures, figurative patterns, etc. dominant types), text trademark images (pure text, font design, etc. text dominant types) and combined trademark images (hybrid arrangement of text and images).

[0054] After collecting the trademark images, the trademark images are labeled and classified according to the Vienna graphic element classification standard, and the classification is accurate to the group (the Vienna classification includes three levels of department, class and group). The text description constructed by this method is more in line with the trademark examination rules, and can simplify the model learning difficulty.

[0055] The text in the trademark image-text pair constructed in this step is composed of the graphic element name in the Vienna Classification Table and the textual information contained in the image. This method is closer to the similarity determination rules in the Trademark Examination and Adjudication Guidelines, and the unified naming rules can make the model easier to learn.

[0056] S2: training a multi-modal representation generation model according to the trademark image-text pair to obtain a trained multi-modal representation generation model;

[0057] Specifically, in this embodiment, the multi-modal representation generation model includes a visual coding module and a cross-modal generation module. The visual coding module uses a CLIP pre-trained Vision Transformer (ViT) model as an image feature extractor. Its advantage lies in that it can realize unified representation of image-text semantic space through contrastive learning, ensuring the consistency of visual features and text description in the hidden space, and laying a foundation for subsequent cross-modal interaction. The cross-modal generation module uses a pre-trained large language model based on a decoder architecture (for example: LLaMA3 8B). This model uses the autoregressive mechanism in the Transformer architecture to perform text generation tasks.

[0058] Please refer to Figure 3 In this step, the multi-modal representation generation model is trained according to the trademark image-text pair using a text-image pair supervised learning strategy, which includes:

[0059] S201: extracting a high-dimensional visual feature vector of the trademark image using the visual coding module, and converting the guiding text instructions into a text vector sequence through a tokenizer;

[0060] In this step, the trademark image is input into the ViT encoder to extract a high-dimensional visual feature vector, and the guiding text instructions (such as "describe this image") are converted into a text vector sequence through a tokenizer.

[0061] S202: aligning the dimensions of the high-dimensional visual feature vector and the text vector sequence through a linear projection layer, and then splicing and fusing them to obtain a joint feature according to the trademark image-text pair;

[0062] In order to realize the feature interaction between visual and language modalities, this step aligns the dimensions of the two types of feature vectors (high-dimensional visual feature vector and text vector sequence) through a linear projection layer, and then splices and fuses them to obtain a joint feature.

[0063] S203: inputting the joint feature into the cross-modal generation module to generate a complete description of the graphic elements and text elements corresponding to the trademark image.

[0064] In this step, the multi-modal representation generation model is trained according to the trademark image-text pair. The training target is to take the integrity description of the graphic elements and text elements corresponding to the trademark image as the supervision signal. Through the end-to-end progressive fine-tuning strategy, the model parameters are optimized.

[0065] In this step, the multi-modal representation generation model is trained according to the trademark image-text pair. Through joint optimization of the text generation loss (cross entropy) and the feature comparison loss (InfoNCE), the goal of multi-task learning is achieved, and the loss function is:

[0066]

[0067] Among them,

[0068]

[0069]

[0070] , represents a hyperparameter, in this embodiment, , , L ce represents the difference between the model prediction distribution and the real distribution, N represents the batch size, C represents the size of the vocabulary, and y ic represents the true label of the i-th sample belonging to the class C, represents the probability predicted by the model, represents the contrastive learning loss, represents the positive sample corresponding to x, represents the negative sample corresponding to x.

[0071] In this step, the trained model takes the last valid token of the last hidden state as the multi-modal representation. This fusion mechanism inherits the dual-modal understanding ability of the pre-trained model, and at the same time enhances the generation effect of the trademark image features through parameter updating.

[0072] It should be noted that the training of the multi-modal representation generation model adopts a progressive training strategy, and step S2 is the first stage of the training of the multi-modal representation generation model. In this stage, the ViT parameters are frozen and only the projection layer and the decoder (the pre-trained large language model of the decoder architecture) are fine-tuned.

[0073] The reason for fine-tuning the multi-modal representation generation model using trademark image-text data instead of using the original model with pre-trained parameters is that the original model with pre-trained parameters is trained on general data. After fine-tuning on trademark data, the model can better understand the characteristics of trademark images.

[0074] The pre-trained large language model is used to replace the text encoder in the CLIP, which can improve the understanding of the semantics in the graph, and the generated feature vector can better fuse the graphical elements and text information in the graph.

[0075] S3: generating a trademark image feature vector by using the trained multi-modal representation generation model, and inputting the vector into a database to obtain an initial trademark retrieval library;

[0076] Specifically, the multi-modal representation generation model is used to encode the trademark image, and the milvus vector database is inputted to obtain an initial trademark retrieval library, which is used for subsequent secondary fine-tuning model.

[0077] S4: obtaining trademark rejection data, selecting trademarks similar to the trademark rejection data from the initial trademark retrieval library as negative samples, and selecting trademarks corresponding to the trademark rejection data in the rejection data as positive samples, and using the adapter technology and the triple contrast learning method to fine-tune the trained multi-modal representation generation model to obtain an optimized multi-modal representation generation model;

[0078] Specifically, according to the characteristics of trademark retrieval, the trademark rejection data is collected, and the adapter enhancement technology is used to freeze the parameters of the above multi-modal representation generation model, and by introducing a small amount of additional parameters (adapter module parameters), the multi-modal representation generation model learns the potential trademark similarity determination rules in the trademark rejection data while ensuring the ability of trademark image-text fusion remains unchanged.

[0079] Please refer to Figure 4 , step S4 specifically includes:

[0080] S401: obtaining trademark rejection data, selecting trademarks similar to the trademark rejection data from the initial trademark retrieval library as negative samples, and selecting trademarks corresponding to the trademark rejection data in the rejection data as positive samples;

[0081] S402: inputting the trademark rejection data (i.e. the rejected trademark image), the positive sample and the negative sample into the trained multi-modal representation generation model to generate a plurality of feature vectors (multi-modal representation);

[0082] S403: inputting the generated plurality of feature vectors into the adapter module to obtain a plurality of new feature vectors, i.e. the positive sample feature vector, the rejected trademark feature vector and the negative sample feature vector;

[0083] S404: using the triple contrast learning method to compare the positive sample feature vector, the rejected trademark feature vector and the negative sample feature vector, and updating the adapter module parameters.

[0084] In this step, Anchor in the triple (Anchor, Positive, Negative) represents a random trademark image, Positive represents the corresponding rejected trademark, and Negative represents the corresponding similar trademark of Anchor in the initial trademark retrieval library.

[0085] The triple contrast learning strategy is: for a rejected trademark image, find its corresponding trademark image in the rejection data as its positive sample, find the top 20 most similar trademarks to the rejected trademark image in the initial trademark retrieval library as its negative sample set, and then according to its training process, from easy to difficult, take out the corresponding difficulty as negative samples from the negative sample set.

[0086] For example: the 20th ranked negative sample is taken in the first two rounds, and the 19th ranked negative sample is taken in the 3rd to 4th rounds, and the training loss function is:

[0087]

[0088] Where A represents the feature vector of Anchor after adapter, i.e. the positive sample feature vector, P represents the feature vector of Positive after adapter, i.e. the rejected trademark feature vector, and N represents the feature vector of Negative after adapter, i.e. the negative sample feature vector, and respectively represent the distance between the anchor sample and the positive sample, and the distance between the anchor sample and the negative sample in the embedding space, which is specifically defined as , margin represents a pre-set threshold value for controlling the difference between the positive sample and the negative sample, in this embodiment, margin = 0.2, and it is hoped that the distance between the anchor sample and the negative sample is greater than the distance between the anchor sample and the positive sample. The purpose of this is to optimize the model performance by gradually increasing the difficulty of the contrast learning task.

[0089] In this embodiment, the adapter module is generally composed of two feedforward (Feedforward) sublayers, which can be abstracted as a mathematical expression as follows:

[0090]

[0091] Where W1, b1, W2, b2 represent the parameters learned by the feedforward sublayer, and the principle is to project the high-dimensional features of the original input to low-dimensional features, and then restore them to the original feature dimension after training, i.e. the first feedforward sublayer projects the input dimension n (high-dimensional features) to m (low-dimensional features), here m < n, and then in the output stage, through the second feedforward sublayer, the input dimension is restored, and m (low-dimensional features) is mapped back to n (original high-dimensional features).

[0092] It should be noted that step S4 is the second stage of the multi-modal representation generation model training, which needs to be fully fine-tuned to improve the retrieval effect. The two-level technical architecture of the embodiment can achieve step-by-step improvement of the retrieval effect.

[0093] This step uses trademark rejection data to further fine-tune the multi-modal representation generation model using a triple comparison learning method, so that the model implicitly learns trademark similarity rules, making trademark retrieval more accurate.

[0094] S5: Use the optimized multi-modal representation generation model to regenerate the trademark image feature vector and record it in the vector database to obtain the final trademark retrieval library.

[0095] The embodiment uses the last optimized multi-modal representation generation model to re-encode the trademark data and record it in the milvus vector database to obtain the final trademark retrieval library for subsequent trademark retrieval.

[0096] The trademark retrieval library generation method based on the multi-modal representation generation model provided in the embodiment has the following advantages:

[0097] (1) Fine-tune the multi-modal representation generation model using trademark data so that the model can better understand trademark graphic elements and text information. Compared with the trademark retrieval library constructed using the original model, the retrieval effect is improved by 8%;

[0098] (2) Fine-tune the model again using trademark rejection data to make the model implicitly learn trademark retrieval rules, better meet the retrieval expectations of real scenarios, and further improve the overall retrieval effect of the constructed trademark retrieval library by 5%;

[0099] (3) It can effectively alleviate the core pain points such as cross-modal semantic gap and rule embedding difficulty in existing trademark retrieval systems, and provides a reusable technical paradigm for the deep application of artificial intelligence in the field of intellectual property.

[0100] Embodiment Two

[0101] The embodiment provides a trademark retrieval library generation device based on a multi-modal representation generation model, which comprises:

[0102] A trademark image-text pair construction module is configured to obtain a trademark image, and label and classify the trademark image according to the Vienna graphic element classification standard to obtain a trademark image-text pair. The trademark image includes a graphic trademark image, a text trademark image, and a combined trademark image.

[0103] The multi-modal representation generation model training module is configured to train a multi-modal representation generation model according to the trademark image-text pair, and obtain a trained multi-modal representation generation model; the multi-modal representation generation model comprises a visual coding module and a cross-modal generation module; the training process specifically comprises: extracting a high-dimensional visual feature vector of the trademark image by using the visual coding module, and converting the guiding text instruction into a text vector sequence by using a word segmenter; performing dimension alignment on the high-dimensional visual feature vector and the text vector sequence by using a linear projection layer, and then performing splicing fusion to obtain a joint feature according to the trademark image-text pair; and inputting the joint feature into the cross-modal generation module to generate a completeness description of the graphical elements and the text elements corresponding to the trademark image;

[0104] The initial trademark retrieval library construction module is configured to generate a trademark image feature vector by using the trained multi-modal representation generation model, and record the trademark image feature vector in a vector database to obtain an initial trademark retrieval library.

[0105] The multi-modal representation generation model optimization module is configured to obtain trademark rejection data, select a trademark similar to the trademark rejection data from the initial trademark retrieval library as a negative sample, select a trademark corresponding to the trademark rejection data in the rejection data as a positive sample, and fine-tune the trained multi-modal representation generation model by using an adapter technology and a triple comparison learning method to obtain an optimized multi-modal representation generation model.

[0106] The trademark retrieval library optimization module is configured to regenerate a trademark image feature vector by using the optimized multi-modal representation generation model, and record the trademark image feature vector in the vector database to obtain a final trademark retrieval library.

[0107] The specific implementation content of each module in the trademark retrieval library generation device based on the multi-modal representation generation model can be referred to the limitation of the trademark retrieval library generation method based on the multi-modal representation generation model in the foregoing, and will not be described here.

[0108] Embodiment three

[0109] The embodiment provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the trademark retrieval library generation method based on the multi-modal representation generation model when executing the computer program.

[0110] The technical features of the above embodiments can be combined in any manner (as long as the combination of the technical features does not exist contradictions), in order to make the description simple, not all possible combinations of the technical features in the above embodiments are described; the embodiments not explicitly written should also be considered as the scope of the present disclosure.

Claims

1. A method for generating a trademark search database based on a multimodal representation generation model, characterized in that: include: Step 1: Obtain a trademark image and annotate and classify it according to the Vienna Classification of Graphic Elements to obtain a trademark image-text pair; the trademark image includes a graphic trademark image, a text trademark image, and a combination trademark image; Step 2: training a multimodal representation generation model based on the trademark image-text pair to obtain a trained multimodal representation generation model; the multimodal representation generation model includes a visual encoding module and a cross-modal generation module; The training process specifically includes: using a visual encoding module to extract high-dimensional visual feature vectors of the trademark image, and converting the guiding text instructions into a text vector sequence through a word segmenter; based on the trademark image-text pair, the high-dimensional visual feature vector and the text vector sequence are dimensionally aligned through a linear projection layer, and then spliced ​​and fused to obtain a joint feature; the joint feature is input into a cross-modal generation module to generate a complete description of the graphic elements and text elements corresponding to the trademark image; Step 3: Generate trademark image feature vectors using the trained multimodal representation generation model and enter them into the vector database to obtain the initial trademark search database; Step 4: Obtain trademark rejection data, select trademarks similar to the trademark rejection data from the initial trademark search database as negative samples, and select trademarks corresponding to the trademark rejection data from the rejection data as positive samples, and use adapter technology and triple contrastive learning methods to fine-tune the trained multimodal representation generation model to obtain an optimized multimodal representation generation model; Step 5: Use the optimized multimodal representation generation model to regenerate the trademark image feature vector and enter it into the vector database to obtain the final trademark retrieval library.

2. The method for generating a trademark search database based on a multimodal representation generation model according to claim 1, characterized in that: In step 2, the visual encoding module uses the Vision Transformer model pre-trained by CLIP.

3. The method for generating a trademark search database based on a multimodal representation generation model according to claim 1, characterized in that: In step 2, the cross-modal generation module adopts a pre-trained large language model based on a decoder architecture.

4. The method for generating a trademark search database based on a multimodal representation generation model according to claim 1, characterized in that: In step 2, when training a multimodal representation generation model based on the trademark image-text pair, the loss function is: ， in, ， ， 、 represents the hyperparameter, L CE It represents the difference between the predicted distribution of the model and the true distribution, N represents the batch size, C represents the vocabulary size, and y ic Indicates that the true label of the i-th sample belongs to category C, represents the probability predicted by the model, represents the contrastive learning loss, represents the positive sample corresponding to x, Indicates the negative sample corresponding to x.

5. The method for generating a trademark search database based on a multimodal representation generation model according to claim 1, characterized in that: In step 3, the vector database adopts the milvus vector database.

6. The method for generating a trademark search database based on a multimodal representation generation model according to claim 1, characterized in that: Step 4 specifically includes: Step 401: Obtain trademark rejection data, select trademarks similar to the trademark rejection data from the initial trademark search database as negative samples, and select trademarks corresponding to the trademark rejection data from the rejection data as positive samples; Step 402: Inputting the trademark rejection data, the positive samples, and the negative samples into the trained multimodal representation generation model to obtain multiple multimodal representations; Step 403: Input the plurality of multimodal representations into the adapter module respectively to obtain a positive sample feature vector, a rejected trademark feature vector, and a negative sample feature vector; Step 404: using a triplet contrast learning method to perform contrast learning on the positive sample feature vector, the rejected trademark feature vector, and the negative sample feature vector, and updating the adapter module parameters.

7. The method for generating a trademark search database based on a multimodal representation generation model according to claim 6, characterized in that: In step 404, during the contrastive learning, the loss function is: ， Among them, A represents the positive sample feature vector, P represents the rejected trademark feature vector, and N represents the negative sample feature vector. and They represent the distance between the anchor sample and the positive sample, and between the anchor sample and the negative sample in the embedding space respectively. Margin represents a preset threshold used to control the difference between the positive sample and the negative sample.

8. The method for generating a trademark search database based on a multimodal representation generation model according to claim 6, characterized in that: The adapter module consists of two feed-forward sub-layers.

9. A trademark search database generation device based on a multimodal representation generation model, characterized in that: include: A trademark image-text pair construction module is used to obtain trademark images and annotate and classify the trademark images according to the Vienna Classification of Graphic Elements to obtain trademark image-text pairs; the trademark images include graphic trademark images, text trademark images, and combination trademark images; a multimodal representation generation model training module, configured to train a multimodal representation generation model based on the trademark image-text pair to obtain a trained multimodal representation generation model; The multimodal representation generation model includes a visual encoding module and a cross-modal generation module; The training process specifically includes: using a visual encoding module to extract high-dimensional visual feature vectors of the trademark image, and converting the guiding text instructions into a text vector sequence through a word segmenter; based on the trademark image-text pair, the high-dimensional visual feature vector and the text vector sequence are dimensionally aligned through a linear projection layer, and then spliced ​​and fused to obtain a joint feature; the joint feature is input into a cross-modal generation module to generate a complete description of the graphic elements and text elements corresponding to the trademark image; An initial trademark search library construction module is used to generate trademark image feature vectors using the trained multimodal representation generation model and enter them into the vector database to obtain the initial trademark search library; a multimodal representation generation model optimization module, configured to obtain trademark rejection data, select trademarks similar to the trademark rejection data from the initial trademark search database as negative samples, select trademarks corresponding to the trademark rejection data from the rejection data as positive samples, and fine-tune the trained multimodal representation generation model using adapter technology and triple contrastive learning methods to obtain an optimized multimodal representation generation model; The trademark search library optimization module is used to regenerate trademark image feature vectors using the optimized multimodal representation generation model and enter them into the vector database to obtain the final trademark search library.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images