Trademark multi-modal retrieval method and device based on image and category similarity and storage medium

Abstract

The invention relates to the technical field of image recognition, in particular to a trademark multi-modal retrieval method and device based on image and category similarity and a storage medium. The method comprises the following steps: acquiring key information input to a trademark retrieval system by a user; inputting the to-be-applied trademark information into a pre-trained multi-modal model to obtain a target multi-modal feature; comparing the target multi-modal features in a trademark database to obtain a first similarity with each trademark; selecting a trademark set from a trademark database according to the first similarity; calculating a second similarity between the to-be-applied category information and the category information of each registered trademark; and sorting and displaying each registered trademark according to the second similarity and the first similarity. According to the method, trademark retrieval and sorting are performed in combination with the image similarity and the category similarity, and the registered trademark with clear influence on the trademark to be applied can be accurately displayed.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a trademark multimodal retrieval method, device and storage medium based on image and category similarity. Background Art

[0002] A trademark is a sign used to identify and distinguish the source of goods or services, including words, graphics, letters, numbers, three-dimensional signs, color combinations, etc. A trademark can be any sign that can distinguish the goods of a natural person, legal person or other organization from those of others.

[0003] The trademark search system is a tool for searching trademark information. It provides a variety of functions and operation modes to meet the needs of different users. The trademark search system usually has an intelligent matching function, which can automatically match relevant trademark information according to the keywords or images entered by the user. This greatly improves the accuracy and efficiency of the query. However, the current trademark search system generally simply matches images in the trademark database through image similarity, and searches for images by image, so as to find trademarks with similar images. Since trademarks have different categories, trademark searches based solely on image similarity cannot meet the needs of trademark searches. Summary of the invention

[0004] The purpose of the present invention is to provide a trademark multimodal retrieval method, device and storage medium based on image and category similarity, so as to combine image similarity and category similarity to perform trademark retrieval and sorting, and accurately display registered trademarks that have a clear impact on the trademark to be applied for.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a trademark multimodal retrieval method based on image and category similarity, comprising:

[0007] Acquire key information input by a user into the trademark search system, wherein the key information includes information on categories to be applied for and information on trademarks to be applied for;

[0008] Inputting the trademark information to be applied for into a pre-trained multimodal model to obtain target multimodal features;

[0009] Comparing the target multimodal feature in a trademark database to obtain a first similarity with each trademark;

[0010] According to the first similarity, a trademark set is selected from the trademark database; the trademark set includes multiple registered trademarks;

[0011] Calculating a second similarity between the category information to be applied for and the category information of each registered trademark in the trademark set;

[0012] According to the second similarity and the first similarity, each registered trademark is sorted and displayed.

[0013] In a second aspect, the present invention provides an electronic device, comprising:

[0014] at least one processor, and a memory communicatively coupled to at least one of the processors;

[0015] Wherein, the memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute the above-mentioned trademark multimodal retrieval method based on image and category similarity.

[0016] In a third aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable the computer to execute the above-mentioned trademark multimodal retrieval method based on image and category similarity.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The present invention obtains a set of trademarks with similar multimodal features by inputting key information into the trademark search system, and sorts and displays them according to category similarities and multimodal similarities. By utilizing the category characteristics of trademarks, it is possible to accurately identify registered trademarks that have an impact on the trademark to be applied for.

[0019] 2. The present invention obtains the target multimodal features by inputting the trademark information to be applied for into the pre-trained multimodal model; and obtains the first similarity with each trademark by comparing the target multimodal features in the trademark database. Compared with simple image comparison, multimodal features can make full use of the complementarity between different modalities through multimodal information comparison, thereby improving the accuracy of comparison; moreover, users are allowed to input pictures, text or graphic information in the trademark search system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1It is a flowchart of a trademark multimodal retrieval method based on image and category similarity provided by the present invention;

[0022] Figure 2 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0024] The present invention is further described in detail below with reference to the embodiments.

[0025] Example 1

[0026] Figure 1 This is a flowchart of a trademark multimodal retrieval method based on image and category similarity provided by the present invention. This embodiment is applicable to scenarios where a trademark search system is used to search for trademarks based on key information input by a user, and the retrieved trademarks are displayed. The method can be executed by a trademark search system, which can be composed of software and / or hardware and is generally integrated in an electronic device. Figure 1 As shown, this embodiment provides a trademark multimodal retrieval method based on image and category similarity, comprising the following steps:

[0027] S110. Obtain key information input by the user into the trademark search system.

[0028] When users have a need for trademark search, they can input the unregistered trademark category information (called pending application category information) and pending application trademark information into the trademark search system. Among them, the pending application category information is a category information in the internationally used "International Classification of Goods and Services for Trademark Registration", or a user-defined category, generally in text form. The pending application trademark information can be an image or a paragraph of text (for example, "a blooming flower with 5 petals"), or a combination of images and text.

[0029] S120: Input the trademark information to be applied for into a pre-trained multimodal model to obtain target multimodal features.

[0030] The pre-trained multimodal model is a bootstrapping language-image pre-training model (BLIP) or a contrastive language-image pre-training model (CLIP). The CLIP model uses a contrastive learning method to train visual and text models. The CLIP model maps the representations of images and texts by jointly learning them to the same vector space, thereby achieving cross-modal alignment. In this embodiment, the existing image library and text library can be used to pre-train the CLIP model, and the text, image or graphic information input by the user is input into the trained CLIP model and mapped to a feature vector, called the target multimodal feature. The BLIP model can map images and texts to the same semantic space so that they can correspond and compare with each other in this space. This helps the model understand the association between images and texts. Using the existing image library and text library to pre-train the BLIP model, the BLIP model can learn richer and more accurate image and text representations. In this embodiment, the text, image or graphic information input by the user is input into the trained BLIP model and mapped into a semantic vector, which is called the target multimodal feature.

[0031] S130 , comparing the target multimodal feature in the trademark database to obtain a first similarity with each trademark.

[0032] The trademark database includes multiple registered trademarks of all categories. In order to achieve the comparison of multimodal features, it is necessary to extract multimodal features from each registered trademark in the trademark database in advance. Specifically, each registered trademark (i.e., image) in the trademark database is input into a pre-trained multimodal model to obtain multimodal features. The multimodal model can be a CLIP model or a BLIP model. The image of each registered trademark and the corresponding multimodal features are stored in the trademark database.

[0033] The multimodal feature is actually a feature vector. The similarity calculation method between feature vectors can be used to calculate the first similarity between the target multimodal feature and each multimodal feature in the trademark database. Optionally, the cosine similarity between the target multimodal feature and the multimodal feature of each registered trademark is calculated as the first similarity. Assuming that there are 10,000 trademarks in the trademark database, 10,000 multimodal features and 10,000 first similarities are obtained. The first similarity can range from 0 to 1.

[0034] S140: Select a trademark set from a trademark database according to the first similarity.

[0035] A similarity threshold is preset, for example, 0.6. In this embodiment, if the first similarity with a registered trademark is greater than the similarity threshold, the registered trademark is stored in the trademark collection. If 30 of the 10,000 first similarities have first similarities greater than 0.6, the 30 trademark information are placed in the trademark collection. The trademark collection includes multiple registered trademarks. In some special cases, if a trademark with a first similarity greater than the similarity threshold cannot be found, a prompt of "No search results" will be displayed; if a trademark with a first similarity greater than the similarity threshold is found, the trademark image will be directly displayed.

[0036] S150. Calculate the second similarity between the category information to be applied for and the category information of each registered trademark in the trademark set.

[0037] Optionally, based on a pre-trained word embedding model, a first word embedding vector of the category information to be applied for is obtained; based on the pre-trained word embedding model, a second word embedding vector of the category information of each registered trademark in the trademark set is obtained; and the cosine similarity between the first word embedding vector and the second word embedding vector is calculated as the second similarity.

[0038] Among them, the pre-trained word embedding model is obtained by training on a large-scale text corpus, which enables computer programs to better understand natural language. The word embedding model can be a word2vec model or a BERT (Bidirectional Encoder Representations from Transformers) model. The word2vec model uses a shallow neural network to learn the representation of words by predicting context words. Word2vec includes two main architectures: the skip-meta model and the continuous bag of words (CBOW) model. The skip-meta model assumes that a word can be used to generate the surrounding words in the text sequence; while the continuous bag of words model assumes that the central word is generated based on the context words. The BERT model can be used to generate pre-trained word representations, and these representations take into account context information. For example, the category information "office supplies" to be applied for is input into the word embedding model to obtain the first word embedding vector. The category information of each registered trademark in the trademark collection (which can be the first-level category information, the second-level category information, and the description information of the category) is respectively input into the word embedding model to obtain the second word embedding vector. For example, the category information of the registered trademark includes pigments and paints, and the goods are mainly used in the construction, decoration, coatings and other industries.

[0039] Cosine similarity (also called cosine distance) measures the similarity between two vectors by measuring the cosine value of the angle between their inner product spaces. In this embodiment, the cosine similarity is set to a range of 0 to 1 as the second similarity.

[0040] In addition to using semantic similarity to define the distance between categories, this embodiment also uses trademark case information to define the distance between categories. This is because, with the increasing activity of commercial activities, the number of well-known trademarks, famous trademarks, etc. is increasing, and it may not be possible to know the exact trademarks that are protected across categories. Moreover, there is no quantifiable solution for close or similar categories, resulting in an increasing number of trademark infringement and trademark invalidation cases. Therefore, this embodiment keeps up with the latest similar category judgment standards of the current era, learns the distance between categories from actual trademark cases, and corrects the second similarity, thereby obtaining the similarity between categories determined jointly by semantics + trademark cases.

[0041] Specifically, after obtaining the second similarity, the following four steps are performed:

[0042] Step 1: Obtain the first set of registered trademarks under the category information to be applied for, and the second set of registered trademarks under any category information in the trademark set.

[0043] Step 2: Obtain cross-category invalidation case information and cross-category trademark infringement case information for each trademark in the first registered trademark set and each trademark in the second registered trademark set.

[0044] Step 3: Determine the similarity between categories based on the case closure information in the case information.

[0045] The case closing information is the conclusion of the final instance court on the trademark case, summarizing the two parties of the trademark case, the nature of the case, the conclusion, etc. Optionally, extract keywords from the case closing information, the keywords at least include win, lose, invalid, valid and similar categories; based on the keywords, it is determined that the categories of the two trademarks in the case belong to similar categories and are judged to be infringing or invalid, then the similarity of the two categories is accumulated by 1. On the contrary, if it is determined based on the keywords that the categories of the two trademarks in the case do not belong to similar categories and are judged to be non-infringing or valid, then the similarity of the two categories is reduced by 1. The final similarity between the categories is calculated, that is, the final accumulated value. For example, through a trademark infringement case, the similarity between the 11th category electric blankets and the 24th category quilts, quilt covers, bedspreads, etc. is set to 1.

[0046] Step 4: Modify the second similarity according to the similarity degree.

[0047] Optionally, the numerical values ​​of the similarity between the two categories are added to the second similarity. For example, the second similarity between the 11th category of electric blankets and the 24th category of quilts, quilt covers, bedspreads, etc. is 0.6, and the corrected second similarity is 1.6.

[0048] S160. Sort and display each registered trademark according to the second similarity and the first similarity.

[0049] Specifically, the first similarity and the second similarity are weighted and summed (the weight can be defined according to business needs) to obtain the similarity score between the trademark to be applied for and each registered trademark in the trademark set. According to the size of the similarity score, each registered trademark is sorted and displayed. The higher the ranking score, the higher the ranking, and the higher the priority to be seen by users. In some cases, although the trademark images are very similar, the categories are not similar, and they will be ranked at the back.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. The present invention obtains a set of trademarks with similar multimodal features by inputting key information into the trademark search system, and sorts and displays them according to category similarities and multimodal similarities. By utilizing the category characteristics of trademarks, it is possible to accurately identify registered trademarks that have an impact on the trademark to be applied for.

[0052] 2. The present invention obtains the target multimodal features by inputting the trademark information to be applied for into the pre-trained multimodal model; and obtains the first similarity with each trademark by comparing the target multimodal features in the trademark database. Compared with simple image comparison, multimodal features can make full use of the complementarity between different modalities through multimodal information comparison, thereby improving the accuracy of comparison; moreover, users are allowed to input pictures, text or graphic information in the trademark search system.

[0053] Example 2

[0054] like Figure 2 As shown, this embodiment provides an electronic device, including:

[0055] at least one processor; and

[0056] a memory communicatively connected to at least one of the processors; wherein,

[0057] The memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors to enable at least one of the processors to execute the above-mentioned trademark multimodal retrieval method based on image and category similarity. At least one processor in the electronic device can execute the above-mentioned method, and thus has at least the same advantages as the above-mentioned method.

[0058] Optionally, the electronic device also includes an interface for connecting various components, including a high-speed interface and a low-speed interface. Various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on a memory to display graphical information of a GUI (Graphical User Interface) on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors can be used together with multiple memories, and / or multiple buses can be used together with multiple memories. Similarly, multiple electronic devices can be connected (for example, as a server array, a group of blade servers, or a multi-processor system), and each device provides some necessary operations. Figure 2 A processor 301 is taken as an example.

[0059] The memory 302, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the trademark multimodal retrieval method based on image and category similarity in the embodiment of the present invention. The processor 301 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 302, that is, realizes the above-mentioned trademark multimodal retrieval method based on image and category similarity.

[0060] The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 302 may further include a memory remotely arranged relative to the processor 301, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0061] The electronic device may further include: an input device 303 and an output device 304. The processor 301, the memory 302, the input device 303 and the output device 304 may be connected via a bus or other means. Figure 2 The example of connecting through bus is taken in the following.

[0062] The input device 303 can receive input digital or character information, and the output device 304 can include a display device, an auxiliary lighting device (e.g., LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device can include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0063] Example 3

[0064] This embodiment provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable the computer to execute the above method. The computer instructions on the computer-readable storage medium are used to enable the computer to execute the above method, and thus have at least the same advantages as the above method.

[0065] The medium in the present invention may adopt any combination of one or more computer-readable media. The medium may be a computer-readable signal medium or a computer-readable storage medium. The medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of the medium (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, the medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0066] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0067] The program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF (Radio Frequency), etc., or any suitable combination of the above.

[0068] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0069] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0070] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A trademark multimodal retrieval method based on image and category similarity, characterized in that: include: Acquire key information input by a user into the trademark search system, wherein the key information includes information on categories to be applied for and information on trademarks to be applied for; Inputting the trademark information to be applied for into a pre-trained multimodal model to obtain target multimodal features; Comparing the target multimodal feature in a trademark database to obtain a first similarity with each trademark; According to the first similarity, a trademark set is selected from the trademark database; the trademark set includes multiple registered trademarks; Calculating a second similarity between the category information to be applied for and the category information of each registered trademark in the trademark set; According to the second similarity and the first similarity, each registered trademark is sorted and displayed.

2. The method according to claim 1, characterized in that The pre-trained multimodal model is a guided language image pre-trained model or a comparative language image pre-trained model.

3. The method according to claim 2, characterized in that The trademark database pre-stores the multimodal features of each registered trademark; Comparing the target multimodal feature in the trademark database to obtain a first similarity with each trademark includes: The cosine similarity between the target multimodal feature and the multimodal feature of each registered trademark is calculated as the first similarity.

4. The method according to claim 3, characterized in that: Selecting a trademark set from the trademark database according to the first similarity includes: Pre-set similarity threshold; If the first similarity with a registered trademark is greater than the similarity threshold, the registered trademark is stored in the trademark set.

5. The method according to claim 4, characterized in that Calculating the second similarity between the category information to be applied for and the category information of each registered trademark in the trademark set, including: According to the pre-trained word embedding model, obtain the first word embedding vector of the category information to be applied for; Obtaining a second word embedding vector of the category information of each registered trademark in the trademark set according to the pre-trained word embedding model; Calculate the cosine similarity between the first word embedding vector and the second word embedding vector as the second similarity.

6. The method according to claim 5, characterized in that After calculating the cosine similarity between the first word embedding vector and the second word embedding vector as the second similarity, the method further includes: Obtain the first set of registered trademarks under the category information to be applied for, and the second set of registered trademarks under any category information in the trademark set; Obtain cross-category invalidation case information and cross-category trademark infringement case information of each trademark in the first registered trademark set and each trademark in the second registered trademark set; Determine the similarity between categories based on the case closing information in the case information; The second similarity is modified according to the similarity degree.

7. The method according to claim 6, characterized in that Based on the case closing information in the case information, determine the similarity between categories, including: Extracting keywords from case closing information, where the keywords at least include win, lose, invalid, valid, and similar categories; If, based on the keywords, it is determined that the categories of the two trademarks in the case are similar and infringement or invalidity is determined, the similarity of the two categories is accumulated by 1; If, based on the keywords, it is determined that the categories of the two trademarks in the case do not belong to similar categories and are judged to be non-infringing or valid, the similarity between the two categories is reduced by 1; Statistical summary of the final similarity between categories.

8. The method according to claim 7, characterized in that According to the second similarity and the first similarity, each registered trademark is sorted and displayed, including: Performing a weighted summation of the first similarity and the second similarity to obtain a similarity score between the trademark to be applied for and each registered trademark in the trademark set; Each registered trademark is sorted and displayed according to the size of the similarity score.

9. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to at least one of the processors; Wherein, the memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute the trademark multimodal retrieval method based on image and category similarity as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The medium stores computer instructions, and the computer instructions are used to enable the computer to execute the trademark multimodal retrieval method based on image and category similarity as described in any one of claims 1 to 8.

Images