Trademark search method, device and electronic device

Abstract

The present application provides a trademark search method, apparatus, and electronic device. The method comprises: obtaining training trademark samples, where the training trademark samples are obtained by annotating the training trademarks; using a pre-trained first detection network to extract features from the trademark image to obtain trademark image features; using a pre-trained second detection network to extract features from the trademark text to obtain trademark text features; using a feature fusion detection network to perform feature fusion processing on the trademark image features and trademark text features to obtain fused image features and fused text features; using a loss function to perform backpropagation processing on the pre-trained first detection network, the pre-trained second detection network, and the feature fusion detection network to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network; and using the trained network to search for the trademark to be searched, thereby solving the problem of low trademark search accuracy in the prior art.

Description

Technical Field

[0001] The present application relates to the field of digital image processing, and in particular to a trademark search method, a trademark search device, a computer-readable storage medium, and an electronic device. Background Art

[0002] As my country's emphasis on intellectual property rights has gradually increased, the demand for trademark registration has also been increasing. In the trademark registration process, trademark retrieval is the most commonly used function. Commonly used image-text retrieval technology solutions mostly calculate the features of images and texts separately, and then add certain constraints (loss functions) to these two features for model training, without considering the correlation between images and texts. In the field of trademark retrieval, most of the methods are image-based or text-based. Even if commonly used image-text retrieval technology solutions are applied to the field of trademark retrieval, accurate retrieval of trademarks cannot be achieved. Summary of the Invention

[0003] The main purpose of this application is to provide a trademark search method, a trademark search device, a computer-readable storage medium, and an electronic device to solve the problem of low accuracy of trademark search in the prior art.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a trademark retrieval method is provided, including: obtaining training trademark samples, wherein the training trademark samples are obtained by annotating training trademarks, the training trademark samples include trademark images and trademark texts, and the trademark texts include trademark text descriptions and trademark label texts; using a pre-trained first detection network to perform feature extraction on the trademark images to obtain trademark image features; using a pre-trained second detection network to perform feature extraction on the trademark text to obtain trademark text features; using a feature fusion detection network to perform feature fusion processing on the trademark image features and the trademark text features to obtain fused image features and fused text features; using a loss function to perform back propagation processing on the pre-trained first detection network, the pre-trained second detection network, and the feature fusion detection network to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network; and using the trained first detection network, the trained second detection network, and the trained feature fusion detection network to retrieve the trademark to be retrieved.

[0005] Furthermore, a loss function is used to perform back propagation processing on the pre-trained first detection network, the pre-trained second detection network and the feature fusion detection network to obtain the trained first detection network, the trained second detection network and the trained feature fusion detection network, including: using a first image-text adversarial loss function to perform back propagation processing on the pre-trained first detection network and the pre-trained second detection network; using a bidirectional triplet loss function and a second image-text adversarial loss function to perform back propagation processing on the feature fusion detection network, the pre-trained first detection network and the pre-trained second detection network to obtain the trained first detection network, the trained second detection network and the trained feature fusion detection network.

[0006] Furthermore, the bidirectional triplet loss function is expressed as: L1 = γL 11 +(1-γ)L 12 , where L1 represents the bidirectional triplet loss, L 11 represents the triplet loss of image retrieval text, L 12 represents the triplet loss of text retrieval image, and γ is a coefficient that balances the triplet loss of the image retrieval text and the triplet loss of the text retrieval image.

[0007] Furthermore, the trained first detection network, the trained second detection network and the trained feature fusion detection network are used to search the trademark to be retrieved, including: using the trained first detection network, the trained second detection network and the trained feature fusion detection network to process each trademark in the trademark library to obtain a trademark library feature set; using the trained first detection network, the trained second detection network and the trained feature fusion detection network to process the trademark to be retrieved to obtain features of the trademark to be detected, wherein the features of the trademark to be detected include fused image features and fused text features; comparing the features of the trademark to be detected with the features in the trademark library feature set to obtain a search result for the trademark to be detected.

[0008] Furthermore, the features of the trademark to be detected are compared with the features in the feature set of the trademark library to obtain the search results of the trademark to be detected, including: obtaining the Euclidean distance between the features of the trademark to be detected and each feature in the feature set of the trademark library; determining the reciprocal of the Euclidean distance as the feature similarity; and determining the trademark corresponding to the feature in the trademark library with the largest feature similarity as the trademark most similar to the trademark to be detected.

[0009] Furthermore, the trademark text of the trademark in the trademark database is an OCR (Optical Character Recognition) text or a trademark label text, and the trademark text of the trademark to be detected is a query text or an OCR text.

[0010] Furthermore, the feature fusion detection network includes a first multi-layer perceptron, a second multi-layer perceptron and a third multi-layer perceptron, and the feature fusion detection network is used to perform feature fusion processing on the trademark image features and the trademark text features to obtain fused image features and fused text features, including: using the first multi-layer perceptron to perform dimensionality reduction processing on the trademark image features to obtain reduced-dimensional trademark image features; inputting the reduced-dimensional trademark image features and the trademark text features into the second multi-layer perceptron for feature extraction to obtain process trademark image features and process trademark text features; inputting the process trademark image features and the process trademark text features into the third multi-layer perceptron for feature extraction to obtain the fused image features and the fused text features.

[0011] According to another aspect of the present application, a trademark search device is provided, comprising an acquisition unit, a first feature extraction unit, a second feature extraction unit, a fusion unit, a processing unit, and a retrieval unit, wherein the acquisition unit is configured to acquire training trademark samples, wherein the training trademark samples are obtained by annotating training trademarks, and the training trademark samples include trademark images and trademark texts, and the trademark texts include trademark text descriptions and trademark label texts; the first feature extraction unit is configured to use a pre-trained first detection network to perform feature extraction on the trademark images to obtain trademark image features; the second feature extraction unit is configured to use a pre-trained second detection network to perform feature extraction on the trademark text to obtain trademark text features; the fusion unit is configured to use a feature fusion detection network to perform feature fusion processing on the trademark image features and the trademark text features to obtain fused image features and fused text features; the processing unit is configured to use a loss function to perform back propagation processing on the pre-trained first detection network, the pre-trained second detection network, and the feature fusion detection network to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network; and the retrieval unit is configured to use the trained first detection network, the trained second detection network, and the trained feature fusion detection network to search for a trademark to be searched.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0013] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.

[0014] Applying the technical solution of the present application, in the trademark retrieval method, first, a training trademark sample is obtained, wherein the training trademark sample is obtained by annotating the training trademark, and the training trademark sample includes a trademark image and a trademark text, and the trademark text includes a trademark text description and a trademark label text; then, a pre-trained first detection network is used to perform feature extraction on the trademark image to obtain trademark image features; then, a pre-trained second detection network is used to perform feature extraction on the trademark text to obtain trademark text features; then, a feature fusion detection network is used to perform feature fusion processing on the trademark image features and the trademark text features to obtain fused image features and fused text features; then, a loss function is used to perform back propagation processing on the pre-trained first detection network, the pre-trained second detection network, and the feature fusion detection network to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network; finally, the trained first detection network, the trained second detection network, and the trained feature fusion detection network are used to retrieve the trademark to be retrieved. In this method, the training trademark samples are obtained by annotating the training trademarks, so the feature description of the training trademark samples is more accurate. The first detection network is used to obtain the trademark image features, the second detection network is used to obtain the trademark text features, and then the fusion detection network is used to obtain the fused image features and fused text features, which can achieve a deeper fusion of image features and text features. Finally, the loss function is used to train each detection network to eliminate the modal differences between the image and text. The trained first detection network, the trained second detection network and the trained feature fusion detection network are then used to search the trademark to be retrieved, which can achieve accurate retrieval of the trademark, thereby solving the problem of low accuracy of trademark retrieval in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0016] Figure 1 A flowchart of a trademark search method according to an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram showing different types of patterns according to an embodiment of the present application;

[0018] Figure 3 A flowchart of a trademark search method according to another embodiment of the present application is shown;

[0019] Figure 4 A schematic diagram of a first multilayer perceptron, a second multilayer perceptron, and a third multilayer perceptron in a feature fusion detection network according to an embodiment of the present application is shown;

[0020] Figure 5 A flowchart of a trademark search method according to another embodiment of the present application is shown;

[0021] Figure 6 A schematic diagram showing back propagation processing using a loss function according to an embodiment of the present application is shown;

[0022] Figure 7 A schematic diagram of a gradient reversal method according to an embodiment of the present application is shown;

[0023] Figure 8 A flowchart of a trademark search method according to another embodiment of the present application is shown;

[0024] Figure 9 A schematic diagram showing searching for a trademark to be detected according to an embodiment of the present application is shown;

[0025] Figure 10 A flowchart of a trademark search method according to another embodiment of the present application is shown;

[0026] Figure 11 A flowchart of a trademark search device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element or intervening elements may be present. Moreover, in the specification and claims, when it is described that an element is "connected to" another element, the element may be "directly connected to" the other element or "connected to" the other element through a third element.

[0031] As introduced in the background technology, the accuracy of trademark retrieval in the prior art is low. To solve the above problem, the embodiments of the present application provide a trademark retrieval method, a trademark retrieval device, a computer-readable storage medium, and an electronic device.

[0032] According to an embodiment of the present invention, a trademark search method is provided.

[0033] Figure 1 Flowchart of the trademark search method according to the embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0034] Step S101: obtaining training trademark samples. The training trademark samples are obtained by annotating training trademarks. The training trademark samples include trademark images and trademark texts. The trademark texts include trademark text descriptions and trademark label texts.

[0035] In one optional solution, the training trademarks can be marked according to the characteristics of the trademark images. Figure 2 As shown in Figures 1 and 2 in the , the trademark pattern only has a pattern without a text description, so the annotator needs to provide a text description. For example, Figure 1 can be described as "a check mark" and Figure 2 can be described as "four connected circles"; Figure 2 As shown in Figure 3, the trademark pattern not only has a pattern but also a text mark "ONE TREE", so the annotator also annotates the text description: "a tree"; Figure 2As shown in Pattern 4, when the trademark pattern is all text, the annotator takes all the text as its description: "SMILE Smile". After marking according to the above marking principle, paired markings of trademark images and texts can be obtained as training data, and the following format can be obtained: <Text description: Pattern label>. For example, Pattern 1 is <A tick: Brand A>, Pattern 2 is <Four connected circles: Brand B>, Pattern 3 is <A white whale: Brand C>, and Pattern 4 is <SMILE Smile: Brand D>. The above marking method can make the feature description of the training trademark samples more accurate.

[0036] In practical applications, in order to adapt to the situation where there is no text description for the patterns in the trademark library during retrieval, the above format can be randomly replaced with the form of <Pattern label: Pattern label> during training as a data augmentation strategy for model training. This embodiment can use three different input modes of text, image, and image-text combination for trademark retrieval, enriching the user's input experience and providing more input information for retrieval.

[0037] Specifically, for the data source of training trademarks, we can use publicly available data sets in the academic community, such as the large-scale image data set LogDet-3k for Logo (trademark) detection. This data set has approximately 3,000 logo categories, 200,000 high-quality manually annotated logo objects, and 158,652 images; we can also use trademark patterns in the Trademark Database of the China Trademark Network.

[0038] Step S102, use a pre-trained first detection network to extract features from the above trademark image to obtain trademark image features.

[0039] Specifically, the above pre-trained first detection network can be a ResNet50 convolutional neural network pre-trained on the ImageNet data set, which can fully express the high-level semantic information of the image.

[0040] Step S103, use a pre-trained second detection network to extract features from the above trademark text to obtain trademark text features.

[0041] Specifically, the above pre-trained second detection network can be a Bert model pre-trained on Baidu Encyclopedia and Wikipedia, which can fully express the high-level semantic information of the text.

[0042] Step S104, use a feature fusion detection network to perform feature fusion processing on the above trademark image features and the above trademark text features to obtain fused image features and fused text features.

[0043] Specifically, using a feature fusion detection network to perform feature fusion processing on the above-mentioned trademark image features and the above-mentioned trademark text features can achieve a deeper fusion of image features and text features, shorten the distance between image features and text features, and integrate semantic information and image information, thereby improving the trademark retrieval effect.

[0044] In an optional solution, the feature fusion detection network includes a first multi-layer perceptron, a second multi-layer perceptron, and a third multi-layer perceptron. The feature fusion detection network is used to perform feature fusion processing on the trademark image features and the trademark text features to obtain the fused image features and the fused text features. The specific process is as follows: Figure 3 The specific description is as follows:

[0045] Step 201 : Using the first multi-layer perceptron to perform dimensionality reduction processing on the trademark image features, to obtain reduced-dimensional trademark image features.

[0046] In an optional solution, the image feature dimension of ResNet50 is 2048 dimensions, and the text feature dimension of Bert is 768 dimensions. In order to ensure the consistency of feature dimensions, Figure 4 As shown, the first multi-layer perceptron MLP0 is first used to reduce the dimensionality of the trademark image features to 768 dimensions.

[0047] Step 202: Input the reduced-dimensional trademark image features and the trademark text features into the second multi-layer perceptron to extract features to obtain process trademark image features and process trademark text features.

[0048] Specifically, the trademark image feature dimension and the text feature dimension of Bert after the first multi-layer perceptron MLP0 dimension reduction are both 768 dimensions, as shown in Figure 4 As shown, the two images are input into the second multi-layer perceptron MLP1 to obtain the process trademark image features and process trademark text features respectively.

[0049] In step 203 , the process trademark image features and the process trademark text features are input into the third multi-layer perceptron for feature extraction to obtain the fused image features and the fused text features.

[0050] Specifically, if Figure 4 As shown, the fused image features and the fused text features obtained by the second multi-layer perceptron MLP1 are input into the third multi-layer perceptron MLP2, and the fused image features and the fused text features finally obtained have a deeper fusion level of the fused image features and the fused text features, and the distance between the fused image features and the fused text features is smaller than the above-mentioned trademark image features and trademark text features.

[0051] Step S105: Back propagation processing is performed on the pre-trained first detection network, the pre-trained second detection network, and the feature fusion detection network using a loss function to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network.

[0052] The loss function is used to perform back propagation processing on the pre-trained first detection network, the pre-trained second detection network and the feature fusion detection network to eliminate the modal differences of image text. The specific process is as follows: Figure 5 The specific instructions are as follows:

[0053] Step S301, performing back propagation processing on the pre-trained first detection network and the pre-trained second detection network using a first image-text adversarial loss function;

[0054] In practical applications, in order to make the output features eliminate the inherent properties of image and text modalities, such as Figure 6 As shown, the trademark image features obtained by extracting features from the trademark image using the ResNet50 convolutional neural network are subjected to dimensionality reduction processing, and feature extraction is performed through a multi-layer perceptron to obtain process trademark image features. The process trademark text features obtained by extracting features from the trademark text using the Bert model through a multi-layer perceptron are then back-propagated using the first image-text adversarial loss function L2.

[0055] Step S302, using a bidirectional triplet loss function and a second image-text adversarial loss function to perform backpropagation processing on the above-mentioned feature fusion detection network, the above-mentioned pre-trained first detection network and the above-mentioned pre-trained second detection network to obtain the trained first detection network, the above-mentioned trained second detection network and the above-mentioned trained feature fusion detection network.

[0056] In practical applications, in order to further eliminate the inherent properties of image and text modalities in the output features, such as Figure 6 As shown, a bidirectional triplet loss function L1 and a second image-text adversarial loss function L3 are used to perform backpropagation processing on the feature fusion detection network, the pre-trained first detection network, and the pre-trained second detection network. The combined effect of the bidirectional triplet loss function L1, the first image-text adversarial loss function L2, and the second image-text adversarial loss function L3 can be expressed as a loss function L = L1 + αL2 + βL3, where α and β are the weights of the two adversarial losses, and α can be set to 0.1 and β to 1.

[0057] Specifically, the bidirectional triplet loss function L1 and the second image-text adversarial loss function L3 aim to make the output features eliminate the inherent properties of the image and text modalities as much as possible. Reference adversarial loss functions include generative adversarial network loss and gradient reversal method. The main idea of ​​the gradient reversal method is to add a modality classifier to the feature. This classifier is used to distinguish the source of the feature (from image or text). When training the classifier, the normal gradient backpropagation method is used (the gradient is ), and when the gradient is back-propagated through the feature layer to the feature extractor, the propagated gradient is modified to a negative gradient The role of the negative gradient here is to make the features generated by the feature extractor eliminate the modal characteristics as much as possible. This is a game process. The two sides of the game are the modal classifier and the feature extractor. The interactive clue of the game is gradient propagation. The modal classifier uses positive gradient, while the feature extractor uses negative gradient, such as Figure 7 shown.

[0058] In an optional solution, the above two-way triple loss function is expressed as: L1 = YL 11 +(1-γ)L 12 , where L1 represents the above-mentioned bidirectional triplet loss, L 11 represents the triplet loss of image retrieval text, L 12 represents the triplet loss of text retrieval image, and Y is the coefficient that balances the triplet loss of image retrieval text and the triplet loss of text retrieval image. Specifically,

[0059]

[0060] Where N represents the number of triplets in one training iteration, f I () is the fusion image feature, f T () is the fusion text feature, x i Represents each image sample, x t represents each text sample, m represents the minimum distance between the retrieval sample and the positive sample, and the distance between the specified retrieval sample and the negative sample, a is the identifier of the retrieval sample, p is the identifier of the positive sample, and n is the identifier of the negative sample. This bidirectional ternary loss function can bring the feature distances of text pairs with the same trademark closer and the feature distances of text pairs with different trademarks farther apart.

[0061] Step S106 , using the trained first detection network, the trained second detection network, and the trained feature fusion detection network to search for the trademark to be searched.

[0062] Specifically, after completing the design of the general trademark fusion retrieval training model, combined with the annotation in step S101, the model training process can be started. The entire training process is similar to the general deep learning model training process. The Adam optimizer can be used, and the learning rate is set to 0.001. After waiting for 20 to 30 rounds of model training to converge, the optimal model can be output. The optimal model is the trained first detection network, the trained second detection network, and the trained feature fusion detection network, which are used to retrieve the trademark to be retrieved. The specific process of retrieving the trademark to be retrieved is as follows: Figure 8 The specific instructions are as follows:

[0063] Step 401 : Process each trademark in the trademark library using the trained first detection network, the trained second detection network, and the trained feature fusion detection network to obtain a trademark library feature set.

[0064] In some embodiments, as Figure 9 As shown, the trademark text of the trademark in the trademark database is an OCR text or a trademark label text, and the trademark text of the trademark to be detected is a query text or an OCR text. Figure 9 The models in are the trained first detection network, the trained second detection network and the trained feature fusion detection network.

[0065] In practical applications, for a single trademark, first, the fusion image features f of the pattern can be extracted through the ResNet50 model. I (x i ), then perform OCR text recognition on the image, and combine the OCR recognition result and the label data of the trademark library into a text format <OCR recognition result: trademark label>. If OCR does not recognize the result, the text format is set to <trademark label: trademark label>, and the fusion text feature f of the text is extracted through the Bert model T (x t ); take the weighted mean of the fused image features and the fused text features as the final feature of the trademark, such as: f(x) = 1 / 2(f I (x i )+f T (x t )), then the trademark database feature set is F G ={f1(x),...,f n (x)}.

[0066] In step 402, the trained first detection network, the trained second detection network, and the trained feature fusion detection network are used to process the trademark to be retrieved to obtain features of the trademark to be detected. The features of the trademark to be detected include fused image features and fused text features.

[0067] In practical applications, for the feature extraction of trademarks to be retrieved, different processing methods are adopted according to different input types. If the input is only text, the fused text features are extracted through the Bert model as the final features; if the input is only a pattern, the fused image features of the pattern are extracted through the ResNet50 model. If there is an OCR result for the pattern, the OCR result is formed into the format <OCR result: OCR result>, and the fused text features of the text are extracted through the Bert model. The fused image features and text features are weighted and averaged to obtain the final features of the input pattern. If the input is a combination of text and pattern, the fused text features and image features are extracted according to the above rules respectively, and then these two features are weighted and averaged to obtain the final features of the trademark to be retrieved this time.

[0068] Step 403: Compare the features of the trademark to be detected with the features in the trademark library feature set to obtain the retrieval result of the trademark to be detected.

[0069] Specifically, the feature comparison and sorting process refers to the process of searching for the features of the trademark to be detected in the trademark library feature set. The specific process is as Figure 10 shown and is specifically described as follows:

[0070] Step 501: Obtain the Euclidean distance between the features of the trademark to be detected and each feature in the trademark library feature set;

[0071] Step 502: Determine the reciprocal of the Euclidean distance as the feature similarity;

[0072] Step 503: Determine the trademark corresponding to the feature in the trademark library with the maximum feature similarity as the trademark most similar to the trademark to be detected.

[0073] In some embodiments, first calculate the Euclidean distance between the features of the trademark to be detected and the features of the trademark library feature set respectively, use the reciprocal of the Euclidean distance as the feature similarity, and then sort them according to the similarity from high to low. The trademark samples with high similarity are ranked靠前, and the trademark samples with low similarity are ranked靠后. Finally, find the corresponding trademark pattern sequence according to the similarity sorting result as the final retrieval result.

[0074] In the above trademark retrieval method, first, a training trademark sample is obtained. The above training trademark sample is obtained by annotating the training trademark. The above training trademark sample includes a trademark image and a trademark text. The above trademark text includes a trademark text description and a trademark label text; then, a pre-trained first detection network is used to extract features of the above trademark image to obtain trademark image features; then, a pre-trained second detection network is used to extract features of the above trademark text to obtain trademark text features; then, a feature fusion detection network is used to perform feature fusion processing on the above trademark image features and the above trademark text features to obtain fused image features and fused text features; then, a loss function is used to perform backpropagation processing on the above pre-trained first detection network, the above pre-trained second detection network and the above feature fusion detection network to obtain a trained first detection network, a trained second detection network and a trained feature fusion detection network; finally, the trained first detection network, the trained second detection network and the trained feature fusion detection network are used to retrieve the trademark to be retrieved. In this method, the training trademark samples are obtained by annotating the training trademarks, so the feature description of the training trademark samples is more accurate. The first detection network is used to obtain the trademark image features, the second detection network is used to obtain the trademark text features, and then the fusion detection network is used to obtain the fused image features and fused text features, which can achieve a deeper fusion of image features and text features. Finally, the loss function is used to train each detection network to eliminate the modal differences between the image and text. The trained first detection network, the trained second detection network and the trained feature fusion detection network are then used to search the trademark to be retrieved, which can achieve accurate retrieval of the trademark, thereby solving the problem of low accuracy of trademark retrieval in the existing technology.

[0075] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0076] The present application also provides a trademark search device. It should be noted that the trademark search device of the present application can be used to execute the trademark search method provided in the present application. The trademark search device provided in the present application is introduced below.

[0077] Figure 11 Schematic diagram of a trademark search device according to an embodiment of the present application. Figure 11 As shown, the device includes an acquisition unit 10, a first feature extraction unit 20, a second feature extraction unit 30, a fusion unit 40, a processing unit 50 and a retrieval unit 60, which are specifically as follows:

[0078] The above-mentioned acquisition unit 10 is used to acquire training trademark samples. The above-mentioned training trademark samples are obtained by annotating training trademarks. The above-mentioned training trademark samples include trademark images and trademark texts. The above-mentioned trademark texts include trademark text descriptions and trademark label texts.

[0079] In an optional solution, when annotating the above-mentioned training trademarks, the annotation can be carried out according to different characteristics of the trademark patterns. For example, Figure 2 As shown in Pattern 1 and Pattern 2 in [ ], if the trademark pattern only has a pattern without a text description, then the annotator needs to describe it in text. For example, Pattern 1 can be described as "a tick", and Pattern 2 can be described as "four connected circles"; as Figure 2 As shown in Pattern 3 in [ ], if the trademark pattern not only has a pattern but also has a text logo "ONE TREE", then the annotator also makes a text description annotation: "a tree"; for Figure 2 As shown in Pattern 4 in [ ], when the trademark pattern is all text, then the annotator takes all the text as its description: "SMILE smile". After annotating according to the above annotation principle, paired annotations of trademark images and texts can be obtained as training data, and the following format can be obtained: <text description: pattern label>. For example, for Pattern 1, it is <a tick: Brand A>, for Pattern 2, it is <four connected circles: Brand B>, for Pattern 3, it is <a white whale: Brand C>, and for Pattern 4, it is <SMILE smile: Brand D>. The above annotation method can make the feature description of the training trademark samples more accurate.

[0080] In practical applications, in order to adapt to the situation where there is no text description in the trademark library during retrieval, the above format can be randomly replaced with the form of <pattern label: pattern label> during training as a data augmentation strategy for model training. This embodiment can use three different input modes of text, image, and image-text combination for trademark retrieval, enriching the user's input experience and providing more input information for retrieval.

[0081] Specifically, for the data source of training trademarks, we can use publicly available data sets in the academic community, such as LogDet-3k, which has approximately 3,000 logo categories, 200,000 high-quality manually annotated logo objects, and 158,652 images; we can also use trademark patterns in the trademark library of the China Trademark Network.

[0082] The above-mentioned first feature extraction unit 20 is used to extract features from the above-mentioned trademark image by using a pre-trained first detection network to obtain trademark image features.

[0083] Specifically, the above-mentioned pre-trained first detection network can be a ResNet50 convolutional neural network pre-trained on the ImageNet data set, which can fully express the high-level semantic information of images.

[0084] The second feature extraction unit 30 is configured to extract features from the trademark text using a pre-trained second detection network to obtain trademark text features.

[0085] Specifically, the pre-trained second detection network can be a Bert model pre-trained on Baidu Encyclopedia and Wikipedia, which can fully express the high-level semantic information of the text.

[0086] The fusion unit 40 is configured to perform feature fusion processing on the trademark image features and the trademark text features using a feature fusion detection network to obtain fused image features and fused text features.

[0087] Specifically, using a feature fusion detection network to perform feature fusion processing on the above-mentioned trademark image features and the above-mentioned trademark text features can achieve a deeper fusion of image features and text features, shorten the distance between image features and text features, and integrate semantic information and image information, thereby improving the trademark retrieval effect.

[0088] In an optional solution, the feature fusion detection network includes a first multilayer perceptron, a second multilayer perceptron, and a third multilayer perceptron, and the fusion unit includes a dimensionality reduction module, a first feature extraction module, and a second feature extraction module, as follows:

[0089] The dimensionality reduction module is used to perform dimensionality reduction processing on the trademark image features using the first multi-layer perceptron to obtain the trademark image features after dimensionality reduction.

[0090] In an optional solution, the image feature dimension of ResNet50 is 2048 dimensions, and the text feature dimension of Bert is 768 dimensions. In order to ensure the consistency of feature dimensions, Figure 4 As shown, the first multi-layer perceptron MLP0 is first used to reduce the dimensionality of the trademark image features to 768 dimensions.

[0091] The first feature extraction module is used to input the reduced-dimensional trademark image features and the trademark text features into the second multi-layer perceptron for feature extraction to obtain process trademark image features and process trademark text features.

[0092] Specifically, the trademark image feature dimension and the text feature dimension of Bert after the first multi-layer perceptron MLP0 dimension reduction are both 768 dimensions, as shown in Figure 4 As shown, the two images are input into the second multi-layer perceptron MLP1 to obtain the process trademark image features and process trademark text features respectively.

[0093] The second feature extraction module is used to input the process trademark image features and the process trademark text features into the third multi-layer perceptron for feature extraction to obtain the fused image features and the fused text features.

[0094] Specifically, if Figure 4 As shown, the fused image features and the fused text features obtained by the second multi-layer perceptron MLP1 are input into the third multi-layer perceptron MLP2, and the fused image features and the fused text features finally obtained have a deeper fusion level of the fused image features and the fused text features, and the distance between the fused image features and the fused text features is smaller than the above-mentioned trademark image features and trademark text features.

[0095] The processing unit 50 is used to perform back propagation processing on the pre-trained first detection network, the pre-trained second detection network and the feature fusion detection network using a loss function to obtain the trained first detection network, the trained second detection network and the trained feature fusion detection network.

[0096] The processing unit can eliminate the modal differences of the image text. The processing unit includes a first processing module and a second processing module, which are specifically as follows:

[0097] The first processing module is used to perform back propagation processing on the pre-trained first detection network and the pre-trained second detection network using a first image-text adversarial loss function;

[0098] In practical applications, in order to make the output features eliminate the inherent properties of image and text modalities, such as Figure 6 As shown, the trademark image features obtained by extracting features from the trademark image using the ResNet50 convolutional neural network are subjected to dimensionality reduction processing, and feature extraction is performed through a multi-layer perceptron to obtain process trademark image features. The process trademark text features obtained by extracting features from the trademark text using the Bert model through a multi-layer perceptron are then back-propagated using the first image-text adversarial loss function L2.

[0099] The above-mentioned second processing module is used to use a bidirectional triplet loss function and a second image-text adversarial loss function to perform backpropagation processing on the above-mentioned feature fusion detection network, the above-mentioned pre-trained first detection network and the above-mentioned pre-trained second detection network to obtain the trained first detection network, the above-mentioned trained second detection network and the above-mentioned trained feature fusion detection network.

[0100] In practical applications, in order to further eliminate the inherent properties of image and text modalities in the output features, such as Figure 6As shown, a bidirectional triplet loss function L1 and a second image-text adversarial loss function L3 are used to perform backpropagation processing on the feature fusion detection network, the pre-trained first detection network, and the pre-trained second detection network. The combined effect of the bidirectional triplet loss function L1, the first image-text adversarial loss function L2, and the second image-text adversarial loss function L3 can be expressed as a loss function L = L1 + αL2 + βL3, where α and β are the weights of the two adversarial losses, and α can be set to 0.1 and β to 1.

[0101] Specifically, the bidirectional triplet loss function L1 and the second image-text adversarial loss function L3 aim to make the output features eliminate the inherent properties of the image and text modalities as much as possible. Reference adversarial loss functions include generative adversarial network loss and gradient reversal method. The main idea of ​​the gradient reversal method is to add a modality classifier to the feature. This classifier is used to distinguish the source of the feature (from image or text). When training the classifier, the normal gradient backpropagation method is used (the gradient is ), and when the gradient is back-propagated through the feature layer to the feature extractor, the propagated gradient is modified to a negative gradient The role of the negative gradient here is to make the features generated by the feature extractor eliminate the modal characteristics as much as possible. This is a game process. The two sides of the game are the modal classifier and the feature extractor. The interactive clue of the game is gradient propagation. The modal classifier uses positive gradient, while the feature extractor uses negative gradient, such as Figure 7 shown.

[0102] In an alternative solution, the above two-way triple loss function is expressed as: L1 = γL 11 +(1-Y)L 12 , where L1 represents the above-mentioned bidirectional triplet loss, L 11 represents the triplet loss of image retrieval text, L 12 represents the triplet loss of text retrieval image, γ is the coefficient that balances the triplet loss of image retrieval text and the triplet loss of text retrieval image. Specifically,

[0103]

[0104] Where N represents the number of triplets in one training iteration, f I () is the fusion image feature, f T () is the fusion text feature, x i Represents each image sample, x tDenote each text sample, \(m\) represents a minimum interval parameter between the distance between the retrieval sample and the positive sample and the specified distance between the retrieval sample and the negative sample, \(a\) is the flag of the retrieval sample, \(p\) is the flag of the positive sample, and \(n\) is the flag of the negative sample. This bidirectional triplet loss function can make the feature distances of the same trademark text pairs closer and the feature distances of different trademark text pairs farther.

[0105] The above-mentioned retrieval unit 60 is used to retrieve the trademark to be retrieved by using the trained first detection network, the trained second detection network, and the trained feature fusion detection network.

[0106] Specifically, after completing the design of the general trademark fusion retrieval training model, combined with the above annotations, the training process of the model can be started. The entire training process is similar to the training process of a general deep learning model. The Adam optimizer can be used, and the learning rate is set to 0.001. After waiting for the model to converge after 20 - 30 rounds of training, the optimal model can be output. This optimal model is the trained first detection network, the trained second detection network, and the trained feature fusion detection network, which are used to retrieve the trademark to be retrieved. The third processing module, the fourth processing module, and the comparison module of the above retrieval unit are as follows:

[0107] The above third processing module is used to process each trademark in the trademark library by using the trained first detection network, the trained second detection network, and the trained feature fusion detection network, and obtain the trademark library feature set.

[0108] In some embodiments, as Figure 9 shown, the trademark text of the above trademark in the above trademark library is OCR text or trademark label text, and the trademark text of the above trademark to be detected is query text or OCR text, Figure 9 and the model in

[0109] In practical applications, for a single trademark, first, the fused image feature \(f\) I (x i ) of the pattern can be extracted through the ResNet50 model, and then OCR text recognition is performed on the image. The OCR recognition result and the label data of the trademark library are formed into a text format <OCR recognition result: trademark label>. If no result is recognized by OCR, the text format is set to <trademark label: trademark label>. The fused text feature \(f\) T (x t ) of this text is extracted through the Bert model; the fused image feature and the fused text feature are weighted and averaged as the final feature of this trademark, such as: \(f(x)=1 / 2(f\) I (xi ) + f T (x t )), the trademark library feature set is F G = {f1(x),..., f n (x)}.

[0110] The above fourth processing module is used to process the above trademark to be retrieved by using the trained first detection network, the trained second detection network, and the trained feature fusion detection network, and obtain the features of the trademark to be detected. The features of the trademark to be detected include fused image features and fused text features.

[0111] In practical applications, for the feature extraction of the trademark to be retrieved, there are different processing methods according to different input types. If the input is only text, the fused text feature is extracted through the Bert model as the final feature; if the input is only a pattern, the fused image feature of the pattern is extracted through the ResNet50 model. If there is an OCR result for the pattern, the OCR result is formed into the format <OCR result: OCR result>, and the fused text feature of the text is extracted through the Bert model. The fused image feature and the text feature are weighted and averaged to be used as the final feature of the input pattern; if the input is a combination of text and pattern, the fused text feature and the image feature are extracted respectively according to the above rules, and then these two features are weighted and averaged to be used as the final feature of the trademark to be retrieved this time.

[0112] The above comparison module is used to compare the features of the above trademark to be detected with the features in the above trademark library feature set, and obtain the retrieval result of the above trademark to be detected.

[0113] Specifically, the above comparison module includes an acquisition sub-module, a first determination sub-module, and a second determination sub-module, which are specifically described as follows:

[0114] The above acquisition sub-module is used to obtain the Euclidean distance between the features of the above trademark to be detected and each feature in the above trademark library feature set.

[0115] The above first determination sub-module is used to determine the reciprocal of the above Euclidean distance as the feature similarity.

[0116] The above second determination sub-module is used to determine the trademark corresponding to the feature in the above trademark library with the largest above feature similarity as the trademark most similar to the above trademark to be detected.

[0117] In some embodiments, the Euclidean distance between the features of the trademark to be detected and the features of the trademark library feature set is first calculated, and the reciprocal of the Euclidean distance is used as the similarity of the features. Then, the features are sorted according to the similarity, with trademark samples with high similarity ranking higher and trademark samples with low similarity ranking lower. Finally, the corresponding trademark pattern sequence is found as the final search result based on the similarity sorting result.

[0118] The above-mentioned trademark search device is used to obtain training trademark samples through the above-mentioned acquisition unit. The above-mentioned training trademark samples are obtained by annotating the training trademarks. The above-mentioned training trademark samples include trademark images and trademark texts, and the above-mentioned trademark texts include trademark text descriptions and trademark label texts; the above-mentioned first feature extraction unit is used to use a pre-trained first detection network to extract features from the above-mentioned trademark images to obtain trademark image features; the above-mentioned second feature extraction unit is used to use a pre-trained second detection network to extract features from the above-mentioned trademark text to obtain trademark text features; the above-mentioned fusion unit is used to use a feature fusion detection network to perform feature fusion processing on the above-mentioned trademark image features and the above-mentioned trademark text features to obtain fused image features and fused text features; the above-mentioned processing unit is used to use a loss function to perform back propagation processing on the above-mentioned pre-trained first detection network, the above-mentioned pre-trained second detection network and the above-mentioned feature fusion detection network to obtain a trained first detection network, a trained second detection network and a trained feature fusion detection network; the above-mentioned retrieval unit is used to use the trained first detection network, the trained second detection network and the trained feature fusion detection network to retrieve the trademark to be retrieved. The training trademark samples in the device are obtained by labeling the training trademarks, so the feature description of the training trademark samples is more accurate. The first detection network is used to obtain the trademark image features, the second detection network is used to obtain the trademark text features, and then the fusion detection network is used to obtain the fused image features and fused text features, which can achieve a deeper fusion of image features and text features. Finally, the loss function is used to train each detection network to eliminate the modal differences between the image and text. The trained first detection network, the trained second detection network and the trained feature fusion detection network are then used to search the trademark to be retrieved, which can achieve accurate retrieval of the trademark, thereby solving the problem of low accuracy of trademark retrieval in the existing technology.

[0119] The above-mentioned trademark search device includes a processor and a memory. The above-mentioned acquisition unit, first feature extraction unit, second feature extraction unit, fusion unit, processing unit and search unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0120] The processor includes a core, which retrieves the corresponding program unit from the memory. One or more cores can be set, and the problem of low accuracy of trademark search in the existing technology can be solved by adjusting the core parameters.

[0121] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0122] According to an embodiment of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the trademark search method.

[0123] According to an embodiment of the present application, a processor is further provided, and the processor is used to run a program, wherein the trademark search method is executed when the program is run.

[0124] An embodiment of the present invention provides a device, comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are performed:

[0125] Step S101: Acquire training trademark samples. The training trademark samples are obtained by annotating training trademarks. The training trademark samples include trademark images and trademark texts. The trademark texts include trademark text descriptions and trademark label texts.

[0126] Step S102: extracting features from the trademark image using a pre-trained first detection network to obtain trademark image features;

[0127] Step S103: extracting features from the trademark text using a pre-trained second detection network to obtain trademark text features;

[0128] Step S104: using a feature fusion detection network to perform feature fusion processing on the trademark image features and the trademark text features to obtain fused image features and fused text features;

[0129] Step S105, performing backpropagation processing on the pre-trained first detection network, the pre-trained second detection network, and the feature fusion detection network using a loss function to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network;

[0130] Step S106 , using the trained first detection network, the trained second detection network, and the trained feature fusion detection network to search for the trademark to be searched.

[0131] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0132] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initializing at least the following method steps:

[0133] Step S101: Acquire training trademark samples. The training trademark samples are obtained by annotating training trademarks. The training trademark samples include trademark images and trademark texts. The trademark texts include trademark text descriptions and trademark label texts.

[0134] Step S102: extracting features from the trademark image using a pre-trained first detection network to obtain trademark image features;

[0135] Step S103: extracting features from the trademark text using a pre-trained second detection network to obtain trademark text features;

[0136] Step S104: using a feature fusion detection network to perform feature fusion processing on the trademark image features and the trademark text features to obtain fused image features and fused text features;

[0137] Step S105, performing backpropagation processing on the pre-trained first detection network, the pre-trained second detection network, and the feature fusion detection network using a loss function to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network;

[0138] Step S106 , using the trained first detection network, the trained second detection network, and the trained feature fusion detection network to search for the trademark to be searched.

[0139] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0140] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0143] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0144] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0145] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0146] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0147] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0148] 1) In the above trademark retrieval method, first, a training trademark sample is obtained. The above training trademark sample is obtained by annotating the training trademark. The above training trademark sample includes a trademark image and a trademark text. The above trademark text includes a trademark text description and a trademark label text. Then, a pre-trained first detection network is used to extract features of the above trademark image to obtain trademark image features. Then, a pre-trained second detection network is used to extract features of the above trademark text to obtain trademark text features. Then, a feature fusion detection network is used to fuse the above trademark image features and the above trademark text features to obtain fused image features and fused text features. Then, a loss function is used to perform backpropagation processing on the above pre-trained first detection network, the above pre-trained second detection network, and the above feature fusion detection network to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network. Finally, the trained first detection network, the trained second detection network, and the trained feature fusion detection network are used to retrieve the trademark to be retrieved. In this method, the training trademark samples are obtained by annotating the training trademarks, so the feature description of the training trademark samples is more accurate. The first detection network is used to obtain the trademark image features, the second detection network is used to obtain the trademark text features, and then the fusion detection network is used to obtain the fused image features and fused text features, which can achieve a deeper fusion of image features and text features. Finally, the loss function is used to train each detection network to eliminate the modal differences between the image and text. The trained first detection network, the trained second detection network and the trained feature fusion detection network are then used to search the trademark to be retrieved, which can achieve accurate retrieval of the trademark, thereby solving the problem of low accuracy of trademark retrieval in the existing technology.

[0149] 2) In the above trademark retrieval method, first, a training trademark sample is obtained. The above training trademark sample is obtained by annotating the training trademark. The above training trademark sample includes a trademark image and a trademark text. The above trademark text includes a trademark text description and a trademark label text. Then, a pre-trained first detection network is used to extract features of the above trademark image to obtain trademark image features. Then, a pre-trained second detection network is used to extract features of the above trademark text to obtain trademark text features. Then, a feature fusion detection network is used to fuse the above trademark image features and the above trademark text features to obtain fused image features and fused text features. Then, a loss function is used to back-propagate the above pre-trained first detection network, the above pre-trained second detection network, and the above feature fusion detection network to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network. Finally, the trained first detection network, the trained second detection network, and the trained feature fusion detection network are used to retrieve the trademark to be retrieved. In this method, the training trademark samples are obtained by annotating the training trademarks, so the feature description of the training trademark samples is more accurate. The first detection network is used to obtain the trademark image features, the second detection network is used to obtain the trademark text features, and then the fusion detection network is used to obtain the fused image features and fused text features, which can achieve a deeper fusion of image features and text features. Finally, the loss function is used to train each detection network to eliminate the modal differences between the image and text. The trained first detection network, the trained second detection network and the trained feature fusion detection network are then used to search the trademark to be retrieved, which can achieve accurate retrieval of the trademark, thereby solving the problem of low accuracy of trademark retrieval in the existing technology.

[0150] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A trademark search method, characterized in that: include: Obtaining a training trademark sample, wherein the training trademark sample is obtained by annotating the training trademark, the training trademark sample includes a trademark image and trademark text, and the trademark text includes a trademark text description and a trademark label text; extracting features from the trademark image using a pre-trained first detection network to obtain trademark image features; Using a pre-trained second detection network to extract features from the trademark text to obtain trademark text features; Using a feature fusion detection network to perform feature fusion processing on the trademark image features and the trademark text features to obtain fused image features and fused text features; Performing backpropagation processing on the pre-trained first detection network, the pre-trained second detection network, and the feature fusion detection network using a loss function to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network; Using the trained first detection network, the trained second detection network, and the trained feature fusion detection network to search for the trademark to be searched; The pre-trained first detection network, the pre-trained second detection network, and the feature fusion detection network are back-propagated using a loss function to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network, including: Performing backpropagation processing on the pre-trained first detection network and the pre-trained second detection network using a first image-text adversarial loss function; A bidirectional triplet loss function and a second image-text adversarial loss function are used to perform backpropagation processing on the feature fusion detection network, the pre-trained first detection network, and the pre-trained second detection network to obtain a trained first detection network, the trained second detection network, and the trained feature fusion detection network. The bidirectional triplet loss function is expressed as: ,in, represents the bidirectional triplet loss, represents the triplet loss of image retrieval text, represents the triplet loss of text retrieval images, To balance the coefficient of the triplet loss of the image retrieval text and the triplet loss of the text retrieval image, , , N represents the number of triplets in one training iteration, is the fusion image feature, It is the fusion text feature. Represents each image sample, Represents each text sample, m represents a minimum interval parameter between the distance between the retrieval sample and the positive sample and the distance between the specified retrieval sample and the negative sample, a is the flag of the retrieval sample, p is the flag of the positive sample, and n is the flag of the negative sample.

2. The method according to claim 1, characterized in that The trained first detection network, the trained second detection network, and the trained feature fusion detection network are used to search for the trademark to be searched, including: Using the trained first detection network, the trained second detection network, and the trained feature fusion detection network, each trademark in the trademark library is processed to obtain a trademark library feature set; Using the trained first detection network, the trained second detection network, and the trained feature fusion detection network, the trademark to be retrieved is processed to obtain features of the trademark to be detected, wherein the features of the trademark to be detected include fused image features and fused text features; The features of the trademark to be detected are compared with the features in the trademark library feature set to obtain a search result of the trademark to be detected.

3. The method according to claim 2, characterized in that Comparing the features of the trademark to be detected with the features in the trademark database feature set to obtain a search result for the trademark to be detected, including: Obtaining the Euclidean distance between the feature of the trademark to be detected and each feature in the trademark library feature set; Determine the reciprocal of the Euclidean distance as the feature similarity; The trademark corresponding to the feature in the trademark database with the greatest feature similarity is determined as the trademark most similar to the trademark to be detected.

4. The method according to claim 2, characterized in that The trademark text of the trademark in the trademark database is an OCR text or a trademark label text, and the trademark text of the trademark to be detected is a query text or an OCR text.

5. The method according to claim 1, characterized in that The feature fusion detection network includes a first multilayer perceptron, a second multilayer perceptron, and a third multilayer perceptron. The feature fusion detection network is used to perform feature fusion processing on the trademark image features and the trademark text features to obtain fused image features and fused text features, including: Using the first multi-layer perceptron to perform dimensionality reduction processing on the trademark image features to obtain reduced-dimensional trademark image features; Inputting the reduced-dimensional trademark image features and the trademark text features into the second multi-layer perceptron for feature extraction to obtain process trademark image features and process trademark text features; The process trademark image features and the process trademark text features are input into the third multi-layer perceptron for feature extraction to obtain the fused image features and the fused text features.

6. A trademark search device, characterized in that: include: an acquiring unit, configured to acquire a training trademark sample, wherein the training trademark sample is obtained by annotating the training trademark, the training trademark sample includes a trademark image and trademark text, and the trademark text includes a trademark text description and a trademark label text; a first feature extraction unit, configured to extract features from the trademark image using a pre-trained first detection network to obtain trademark image features; a second feature extraction unit, configured to extract features from the trademark text using a pre-trained second detection network to obtain trademark text features; a fusion unit, configured to perform feature fusion processing on the trademark image features and the trademark text features using a feature fusion detection network to obtain fused image features and fused text features; a processing unit, configured to perform backpropagation processing on the pre-trained first detection network, the pre-trained second detection network, and the feature fusion detection network using a loss function to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network; a retrieval unit, configured to use the trained first detection network, the trained second detection network, and the trained feature fusion detection network to search for a trademark to be searched; The processing unit includes a first processing module and a second processing module, wherein the first processing module is used to perform back propagation processing on the pre-trained first detection network and the pre-trained second detection network using a first image-text adversarial loss function; the second processing module is used to perform back propagation processing on the feature fusion detection network, the pre-trained first detection network, and the pre-trained second detection network using a bidirectional triplet loss function and a second image-text adversarial loss function to obtain a trained first detection network, a trained second detection network, and a trained feature fusion detection network. The bidirectional triplet loss function is expressed as: ,in, represents the bidirectional triplet loss, represents the triplet loss of image retrieval text, represents the triplet loss of text retrieval images, To balance the coefficient of the triplet loss of the image retrieval text and the triplet loss of the text retrieval image, , , N represents the number of triplets in one training iteration, is the fusion image feature, It is the fusion text feature. Represents each image sample, Represents each text sample, m represents a minimum interval parameter between the distance between the retrieval sample and the positive sample and the distance between the specified retrieval sample and the negative sample, a is the flag of the retrieval sample, p is the flag of the positive sample, and n is the flag of the negative sample.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 5.

8. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method according to any one of claims 1 to 5.

Images