Trademark pattern matching method and corresponding device, equipment and medium

Abstract

The present application discloses a trademark pattern matching method and its corresponding device, equipment, and medium, the method comprising: obtaining the image features of the trademark pattern contained in the image to be compared, constructing a feature matrix to be compared, wherein each row vector stores the image features of a single trademark pattern; obtaining a compared feature matrix of a preset compared trademark library, wherein each row vector stores the image features of a single compared trademark in the compared trademark library; calculating similarity based on the feature matrix to be compared and the compared feature matrix to obtain a similarity matrix; screening out elements whose similarity meets preset conditions from the similarity matrix to determine suspected trademark patterns from the trademark pattern that have a similarity association with the compared trademark, and matching each suspected trademark pattern with the first several compared trademarks that are most similar to it. The present application implements the calculation and judgment of trademark approximate matching based on matrix operations, has high calculation efficiency, rapid response, and low overhead, and is suitable for identifying suspected trademark infringement information from product images.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of e-commerce information technology, and in particular to a trademark pattern matching method and its corresponding device, equipment, and medium. Background Art

[0002] Intellectual property issues are becoming increasingly prominent. Trademark searches are often required in different stages of the trademark life cycle, such as application, examination, review, and litigation. Among them, similarity searches of trademark patterns have always been a technical difficulty. In the trademark enforcement stage, a typical application area is the e-commerce field, which often provides product information in the form of product images, and product images usually also contain trademark images.

[0003] E-commerce platforms have a lot of product trademark infringement problems, especially the brand infringement caused by the arbitrary use of trademarks without the authorization of the trademark owner, which will bring great legal risks. Although the platform can apply the "safe harbor" principle to exempt from liability, it is not as efficient as timely discovering suspected infringements in practice. Therefore, how to effectively and quickly discover infringement phenomena and reduce the number of infringing products on the platform is an important issue that needs to be solved urgently.

[0004] Patent application No. CN106530194A published on March 22, 2017 discloses a method and device for detecting images of suspected infringing products. This solution alleviates the problems of slow review, limited detection accuracy and recall based on manual rule review to a certain extent, but it also has some problems:

[0005] For example, in the logo detection part, the solution uses the logo pattern traversal in the pre-established logo pattern library to match the product image to be identified, and requires to know the category of the product image to be identified in advance. Only when the matching degree exceeds a certain threshold is it considered that the corresponding suspected infringing logo exists in the product image to be identified. This process still requires a large amount of calculation, and because the category of the product image to be identified needs to be known in advance, this further depends on the accuracy of the category judgment of the product image to be identified; in addition, if a logo involves categories in different fields, it will also interfere with the final trademark judgment. A typical example is the YAMAHA brand, which involves products in multiple different categories such as piano and motorcycle manufacturing. Simply detecting trademarks under categories may result in missed detection problems; in addition, the solution uses a variety of local feature algorithms to extract relevant feature points. Fewer feature points are extracted for relatively blurred images, and relevant feature points may not be accurately extracted for targets with smooth edges, which may lead to misjudgments.

[0006] This shows that there is still room for improvement in the technology of efficiently and accurately identifying trademark infringement from product images. Summary of the invention

[0007] The purpose of the present application is to provide a trademark pattern matching method and its corresponding device, computer equipment and storage medium to address at least some of the deficiencies in the prior art.

[0008] In order to solve the above technical problems, a technical solution adopted in this application is:

[0009] The present application provides a trademark pattern matching method, comprising the following steps:

[0010] Obtaining image features of the trademark pattern contained in the image to be compared, and constructing a feature matrix to be compared, wherein each row vector stores the image feature of a single trademark pattern;

[0011] Obtaining a compared feature matrix of a preset compared trademark library, wherein each row vector stores an image feature of a single compared trademark in the compared trademark library, and the trademark pattern and the image feature of the compared trademark are both represented as row vectors with the same dimension;

[0012] Calculate similarity based on the feature matrix to be compared and the feature matrix to be compared, and obtain a similarity matrix, wherein each row vector stores the similarity between a single trademark pattern and all the compared trademarks;

[0013] Elements whose similarity meets preset conditions are selected from the similarity matrix to determine suspected trademark patterns that have similarity association with the compared trademark from the trademark patterns, and the first several compared trademarks that are most similar to each suspected trademark pattern are matched.

[0014] In a specific embodiment, obtaining the image features of the trademark pattern contained in the image to be compared and constructing the feature matrix to be compared includes the following steps:

[0015] Get the image to be compared;

[0016] Calling a pre-trained trademark detector to perform trademark pattern detection on the image to be compared, and obtaining one or more trademark patterns contained in the image to be compared;

[0017] The pre-trained feature extractor is called to extract features from each of the trademark patterns to obtain its image features. The image features of each trademark pattern are represented by a row vector, and the row vectors corresponding to all trademark patterns are stored in a feature matrix to be compared.

[0018] In a preferred embodiment, in the step of calling a pre-trained trademark detector to perform trademark pattern detection on a comparison image, in the intermediate process of the trademark detector implementing the detection of the trademark pattern, a convolutional neural network dual-tower structure is applied to output its final image features, and the dual-tower structure includes upsampling and downsampling operations on at least four scales.

[0019] In a preferred embodiment, in the step of obtaining the compared feature matrix of a preset compared trademark library, the compared trademark in the compared trademark library is taken as input, the pre-trained feature extractor is called to extract the image features of the compared trademark, and the image features are normalized into row vectors and added to the compared feature matrix.

[0020] In a specific embodiment, the similarity is calculated based on the feature matrix to be compared and the feature matrix to be compared to obtain a similarity matrix, including the following steps:

[0021] Obtain the transposed matrix of the compared feature matrix;

[0022] Performing matrix multiplication of the feature matrix to be compared and the transposed matrix to obtain a cosine matrix;

[0023] The cosine matrix is ​​converted into a similarity matrix using a preset normalization formula, so that the numerical value of each element in the cosine matrix is ​​normalized to the similarity of each element in the similarity matrix.

[0024] In a specific embodiment, elements whose similarity meets preset conditions are selected from the similarity matrix to determine suspected trademark patterns that have similarity association with the compared trademark from the trademark patterns, and the first several compared trademarks that are most similar to each suspected trademark pattern are determined, including the following steps:

[0025] Traversing each element of the similarity matrix, adding the elements whose similarity meets the preset conditions to the candidate set, determining that the trademark pattern corresponding to the element added to the candidate set and the compared trademark have a similarity association, and determining the trademark pattern as a suspected trademark pattern;

[0026] Clustering all elements in the candidate set according to the suspected trademark pattern, obtaining mapping relationship data including corresponding similarities between each suspected trademark pattern and the compared trademark associated with it;

[0027] The compared trademarks associated with each suspected trademark pattern are sorted according to the similarity, and the first several compared trademarks that are most similar to the suspected trademark pattern are matched.

[0028] In a further extended embodiment, matching the first several compared trademarks that are most similar to the suspected trademark pattern includes the following steps:

[0029] Based on the clustered candidate set, the compared trademarks associated with each suspected trademark pattern are sorted according to the similarity;

[0030] Screen out the first several compared trademarks that are most similar to the suspected trademark pattern to achieve matching, determine the index identifier of the compared trademark that matches the suspected trademark pattern and the similarity between the suspected trademark pattern and the compared trademark;

[0031] The name of the compared trademark is determined by searching a preset index library according to the index identifier, wherein the index library pre-stores mapping relationship data between the name of the compared trademark and the index identifier of the compared trademark in the compared feature library, and the index identifier is correspondingly transmitted between the compared feature matrix, the similarity matrix, and the candidate set;

[0032] Output the mapping relationship data of each suspected trademark pattern and the name of the compared trademark it matches, as well as the similarity between the two.

[0033] In order to solve the above technical problems, another technical solution adopted in this application is:

[0034] The present application provides a trademark pattern matching device, which includes a preparation module to be compared, a preparation module to be compared, a similarity calculation module and a screening and matching module, wherein the preparation module to be compared is used to obtain image features of trademark patterns contained in a to-be-compared image and construct a to-be-compared feature matrix, wherein each row vector stores the image features of a single trademark pattern; the preparation module to be compared is used to obtain a compared feature matrix of a preset compared trademark library, wherein each row vector stores the image features of a single compared trademark in the compared trademark library, and the trademark pattern and the image features of the compared trademark are both represented as row vectors with the same dimension; the similarity calculation module is used to calculate similarity based on the to-be-compared feature matrix and the compared feature matrix to obtain a similarity matrix, wherein each row vector stores the similarity of a single trademark pattern compared with all compared trademarks respectively; the screening and matching module is used to screen out elements whose similarity meets preset conditions from the similarity matrix to determine suspected trademark patterns that have similarity associations with the compared trademarks from the trademark patterns, and match each suspected trademark pattern with the first several compared trademarks that are most similar to it.

[0035] In a specific embodiment, the comparison preparation module includes: a picture receiving submodule, used to obtain the picture to be compared; a trademark detection submodule, used to call a pre-trained trademark detector to perform trademark pattern detection on the picture to be compared, and obtain one or more trademark patterns contained in the picture to be compared; a feature extraction submodule, used to call a pre-trained feature extractor to perform feature extraction on each of the trademark patterns, and obtain its image features. The image feature of each trademark pattern is represented by a row vector, and the row vectors corresponding to all trademark patterns are stored in a feature matrix to be compared.

[0036] In a preferred embodiment, during the intermediate process of detecting the trademark pattern, the trademark detector applies a convolutional neural network dual-tower structure to output the final image features, and the dual-tower structure includes upsampling and downsampling operations at at least four scales.

[0037] In a preferred embodiment, the compared preparation module is configured to take the compared trademark in the compared trademark library as input, call the pre-trained feature extractor to extract the image features of the compared trademark, and normalize the image features into row vectors and add them to the compared feature matrix.

[0038] In a specific embodiment, the similarity calculation module includes: a matrix transposition submodule, used to obtain the transposed matrix of the feature matrix to be compared; a cosine calculation submodule, used to perform matrix multiplication of the feature matrix to be compared with the transposed matrix to obtain a cosine matrix; a similarity normalization submodule, used to apply a preset normalization formula to convert the cosine matrix into a similarity matrix, so as to normalize the numerical values ​​of each element in the cosine matrix to the similarity of each element in the similarity matrix.

[0039] In a specific embodiment, the screening and matching module includes: a traversal and screening submodule, which is used to traverse the elements of the similarity matrix, add the elements whose similarity meets the preset conditions to the candidate set, determine that the trademark pattern corresponding to the element added to the candidate set and the compared trademark have a similarity association, and determine the trademark pattern as a suspected trademark pattern; a clustering and mapping submodule, which is used to cluster all elements in the candidate set according to the suspected trademark pattern, and obtain mapping relationship data including corresponding similarities between each suspected trademark pattern and the compared trademark with which there is a similarity association; a sorting and matching submodule, which is used to sort the compared trademarks associated with each suspected trademark pattern according to the similarity, and match the first several compared trademarks with the lowest similarity to the suspected trademark pattern.

[0040] In a further extended embodiment, the sorting and matching submodule includes: a secondary sorting submodule, which is used to sort the compared trademarks associated with each suspected trademark pattern according to similarity based on the clustered candidate set; a secondary screening submodule, which is used to screen out the first several compared trademarks that are most similar to the suspected trademark pattern to achieve matching, and determine the index identifier of the compared trademark that matches the suspected trademark pattern and the similarity between the suspected trademark pattern and the compared trademark; a secondary query submodule, which is used to query a preset index library according to the index identifier to determine the name of the compared trademark, and the index library pre-stores the mapping relationship data between the name of the compared trademark and the index identifier of the compared trademark in the compared feature library, and the index identifier is correspondingly transmitted between the compared feature matrix, the similarity matrix, and the candidate set; a secondary output submodule, which is used to output the mapping relationship data of each suspected trademark pattern and the name of the compared trademark that matches it and the similarity between the two.

[0041] To solve the above technical problems, the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the trademark pattern matching method.

[0042] In order to solve the above technical problems, an embodiment of the present application further provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the trademark pattern matching method.

[0043] Compared with the prior art, this application has the following advantages:

[0044] On the one hand, the present application constructs a feature matrix to be compared for storing the vectors of the image features of the trademark patterns contained in the product images to be reviewed, and on the other hand, a feature matrix to be compared is pre-constructed for storing the vectors of the image features of the compared goods. The two vectors are unified into the same dimension, and then fast and efficient matrix operations are performed based on the feature matrix to be compared and the feature matrix to be compared, so as to quickly obtain the similarity between the product patterns in the image to be compared and each compared trademark in the compared trademark library with a smaller amount of calculation and memory usage. According to the similarity, the suspected infringing trademark patterns can be further determined, and the compared trademarks and the corresponding similarities that match the suspected infringing trademark patterns in the image to be compared can be selected and associated. During the whole process, the computer operation logic is clear, the operation is efficient, the programming is convenient, the results are obtained quickly, and it is easy to deploy in Internet platforms such as e-commerce to identify trademark infringement behaviors of massive product images, respond to infringement behaviors in a timely manner, and enhance the risk prevention and control capabilities of e-commerce platforms. By extension, the technical solution of the present application is also suitable for trademark retrieval needs, and can be used to assist in the judgment of similar trademarks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 A schematic diagram of the basic process of the trademark pattern matching method of this application;

[0047] Figure 2 A schematic diagram of a flow chart for constructing a feature matrix to be compared for the trademark pattern matching method of the present application;

[0048] Figure 3 A schematic diagram of a process for calculating similarity and obtaining a similarity matrix for the trademark pattern matching method of the present application;

[0049] Figure 4 A schematic diagram of the process of screening suspected trademark patterns by the trademark pattern matching method of this application;

[0050] Figure 5 A schematic diagram of a flow chart for outputting screening results of the trademark pattern matching method of this application;

[0051] Figure 6 A basic structural diagram of the trademark pattern matching device of the present application;

[0052] Figure 7 This is a basic structural block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0054] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0055] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0056] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0057] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0058] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0059] Some technical solutions of this application can be deployed on a cloud server, which can realize data communication connection with business-related servers to coordinate online services, and can also form a logically associated service cluster with other related servers to provide services for related terminal devices such as smart phones, personal computers, third-party servers, etc. The smart phones and personal computers can access the Internet through well-known network access methods, and establish data communication links with the servers of this application to access and use the services provided by the servers.

[0060] For the server, generally, a service engine that provides online services opens a corresponding program interface for remote calls by various terminal devices. The relevant technical solutions suitable for deployment on the server in this application can be implemented in the server in this way.

[0061] The computer program referred to in this application, i.e., the application, is developed in a computer programming language and installed in a computer device, including a server, a terminal device, etc., and is used to implement the relevant functions specified in this application. Unless otherwise specified, it has nothing to do with the development language used.

[0062] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0063] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0064] See also Figure 1The following is a basic flow chart of the trademark pattern matching method of the present application in its typical embodiment. The trademark pattern matching method provided by the present application is programmed as an application program and deployed in a server. It mainly serves the needs of an e-commerce platform to identify whether a product image is suspected of trademark infringement. The method includes the following steps:

[0065] Step S1100: Obtain image features of the trademark pattern contained in the image to be compared, and construct a feature matrix to be compared, wherein each row vector stores the image feature of a single trademark pattern:

[0066] The pictures to be compared can be obtained by providing an access interface for users to call and uploading; they can also be obtained by querying from the product information library of the e-commerce platform through the online service implemented according to this application. Therefore, after the platform applies the technical solution of this application, it can not only find the suspected infringing product pictures through background scanning, but also find them in time when its merchant users upload their product pictures, so as to achieve the purpose of platform risk control.

[0067] The image to be compared may contain one or more trademark patterns, and many image target detection algorithms or tools commonly used by those skilled in the art may be used to identify the trademark patterns therein, preferably using a neural network model based on deep semantic learning, such as YOLO, Faster R-CNN, and SSD (Single Shot Multibox Detector) based on VGG. With the help of such tools, one or more areas corresponding to the trademark patterns are determined in the image to be compared, and the images of each area are extracted to obtain the trademark patterns accordingly.

[0068] The "trademark" attribute of the trademark pattern mainly includes the mark used to indicate the source of the product, any mark that can distinguish the goods of a natural person, legal person or other organization from the goods of others, including text, graphics, letters, numbers, three-dimensional marks, color combinations, etc., and the combination of the above elements, whether registered or unregistered, can constitute the "trademark" referred to in this application. According to the actual needs of the platform's risk control, the concept of "trademark" can also be extended to include objects that are prohibited by law or may have other infringement disputes. When using a neural network model to achieve target detection, the pre-trained model can generally be used to determine the corresponding area for the trademark pattern in the product image, thereby extracting the trademark pattern. Of course, for the purpose of this application, the trademark pattern described is treated as an image for processing regardless of its composition.

[0069] The trademark patterns cut out from the images to be compared are generally unified into the same scale to facilitate standardized processing of extracting image features. In the present application, the image feature extraction of trademark patterns can be realized with the aid of a variety of image feature extraction algorithms or tools, preferably a neural network model based on deep semantic learning, including ordinary CNN networks suitable for extracting image features, residual networks Resnet and their subsequent evolutionary versions such as the Iresnet series, etc., as long as they are suitable for extracting image features of the trademark patterns and obtaining corresponding image features.

[0070] After obtaining the image features corresponding to each trademark pattern, in order to construct a to-be-compared feature matrix corresponding to multiple trademark patterns, the image features corresponding to each trademark pattern are normalized into a row vector of, for example, 2048 dimensions, and the row vector is stored in the to-be-compared feature matrix. Thus, the image features of one or more trademark patterns of the same to-be-compared image are reflected in the to-be-compared feature matrix, and each row vector in the to-be-compared feature matrix corresponds to one of the trademark patterns. Combined with the previous example, assuming that there are B trademark patterns in a picture, the to-be-compared feature matrix is ​​a linear space of B*2048 dimensions.

[0071] Step S1200: Obtain a compared feature matrix of a preset compared trademark library, wherein each row vector stores the image feature of a single compared trademark in the compared trademark library, and the trademark pattern and the image feature of the compared trademark are both represented as row vectors with the same dimension:

[0072] To meet the needs of this application, a compared feature matrix is ​​pre-constructed. The compared feature matrix has the same organizational structure as the compared feature matrix, and contains multiple row vectors. Each row vector is an image feature of the compared trademark. The dimension of this row vector is the same as the dimension of the row vector in the feature matrix to be compared, such as the aforementioned 2048 dimensions.

[0073] In order to construct this compared feature matrix, a compared trademark library is prepared, including multiple image files corresponding to the compared trademarks, that is, a trademark library. These compared trademarks can be pre-selected according to the "trademark" attribute defined in this application, for example, at least they can be collected and extracted from the legitimate products or legitimate trademark patterns of the legitimate merchants of the e-commerce platform itself, and can also be added by the e-commerce platform itself. Generally speaking, the image file corresponding to a compared trademark only stores a single compared trademark to facilitate unified processing.

[0074] In order to ensure that the row vectors in the compared feature matrix have the same dimension as the row vectors in the feature matrix to be compared, the same image feature extraction tool can be used to extract image features of each compared trademark in the compared trademark library as for the feature matrix to be compared. Finally, they are also normalized to, for example, 2048-dimensional row vectors and added to the compared feature matrix.

[0075] After the above processing, the trademark pattern and the image features of the compared trademark are both represented as row vectors with the same dimension, and are stored in the to-be-compared feature matrix and the compared feature matrix respectively.

[0076] To facilitate subsequent indexing, the name of the compared trademark, such as its file name or custom name, and the index identifier of the compared trademark in the compared feature matrix can be extracted and constructed into mapping relationship data, which is stored in an index library so that the name of the compared trademark can be reversely determined based on the index identifier and its image file can be obtained.

[0077] At this point, assuming that the total number of compared products is N, the compared feature matrix is ​​a linear space of N*2048, which has the same row vector length as the feature matrix to be compared of B*2048. The row vector lengths of the two are the same, which facilitates subsequent calculations.

[0078] Step S1300: Calculate similarity based on the feature matrix to be compared and the compared feature matrix to obtain a similarity matrix, wherein each row vector stores the similarity between a single trademark pattern and all compared trademarks:

[0079] This application is based on the principle of cosine similarity algorithm to realize the calculation of the similarity between the feature matrix to be compared and the feature matrix to be compared. According to the cosine similarity principle, the similarity matrix obtained by matrix multiplication of two vector matrices can reflect the similarity between the two.

[0080] It should be noted that the feature matrix to be compared and the feature matrix being compared both store the row components of the trademark pattern and the compared trademark in the same structure. Therefore, when applying the cosine similarity algorithm, the feature matrix to be compared needs to be transposed and then matrix multiplication is performed with the feature matrix to be compared, that is, the B*2048-dimensional feature matrix to be compared and the N*2048-dimensional feature matrix to be compared are matrix multiplied to obtain a B*N cosine matrix. It can be understood that in the cosine matrix, each row vector stores the similarity of a single trademark pattern compared with all the compared trademarks, and each element reflects the similarity between a trademark pattern and a compared trademark. Therefore, in theory, the cosine matrix can also be directly used as the similarity matrix required for subsequent similarity comparison.

[0081] The higher the value of the elements in the cosine matrix, the higher the similarity between the trademark pattern and the compared trademark; the lower the value, the lower the similarity between the trademark pattern and the compared trademark. Although the numerical meaning of the cosine matrix is ​​very intuitive, it is not necessarily a method of calculation. Therefore, according to the actual situation, it can be further normalized to convert the cosine matrix into the final similarity matrix. It can be understood that the similarity matrix obtained after the conversion is still a B*N matrix.

[0082] In another embodiment to be disclosed later in this application, for the convenience of calculation, based on the cosine matrix, the cosine matrix is ​​further converted based on the numerical value of each element using a normalization formula, and each numerical value is projected into the space of [0,1] so as to more intuitively reflect the similarity and facilitate calculation. After this conversion, the meaning of the numerical value in the similarity matrix is ​​converted to when the numerical value is lower, the difference between the trademark pattern and the corresponding compared trademark is smaller and the similarity is higher, and when the numerical value is higher, the difference between the trademark pattern and the corresponding compared trademark is greater and the similarity is lower. The normalization formula described above can be flexibly set by those skilled in the art as long as the aforementioned purpose can be achieved.

[0083] Step S1400: Filter out elements whose similarity meets preset conditions from the similarity matrix to determine suspected trademark patterns that have similarity association with the compared trademark from the trademark patterns, and match the first several compared trademarks that are most similar to each suspected trademark pattern:

[0084] For the similarity in the similarity matrix, technical personnel in this field can flexibly set preset conditions according to the specific meaning reflected by the similarity value, so as to screen out the elements whose similarity meets the preset conditions, so as to screen out the specific trademark patterns and the specific compared trademarks whose similarity is reflected by the elements. The specific trademark patterns thus screened out will be regarded as suspected trademark patterns.

[0085] It should be understood that the preset condition is determined according to the meaning reflected by the similarity value in the similarity matrix, and can be flexibly determined by those skilled in the art according to the specific circumstances of the similarity data distribution, and belongs to empirical data. In the present application, the preset condition is set to whether the similarity in the similarity matrix is ​​lower than a preset threshold value. If it is lower than the preset threshold value, it is deemed to meet the preset condition, otherwise it does not meet the preset condition.

[0086] Specifically, when the values ​​(similarity) of the elements in the similarity matrix are normalized to the space of [0,1], the lower the value, the more similar they are. In fact, it has reflected the difference ratio between the suspected trademark pattern and the specific compared trademark, which is easy to read and highly comparable. Therefore, setting the preset threshold is actually to set the degree of difference between the suspected trademark pattern and the compared trademark, which reflects the degree of similarity. In this case, the preset condition is set to the preset threshold to determine whether the value of the elements in the similarity matrix is ​​lower than the preset threshold. For elements lower than the preset threshold, it means that the difference between the trademark pattern and the compared trademark is small, and the trademark pattern can be determined to be a suspected trademark pattern.

[0087] On the contrary, if the cosine matrix is ​​directly used as the similarity matrix, and the similarity value in the cosine matrix is ​​not normalized, the higher the value of its elements, the higher the similarity. Therefore, when it is screened to adapt to the preset conditions, a preset threshold can be given, and then it is determined whether the similarity is higher than the preset threshold. If it is higher than the preset threshold, it is considered to meet the preset conditions. Those skilled in the art can understand that normalization affects the operation principle of the preset conditions during screening, and this point can be flexibly overcome by those skilled in the art. In any case, those skilled in the art can flexibly use operation means to screen out suspected trademark patterns that meet the preset conditions, so it will not be repeated.

[0088] According to the above-mentioned screening principles, by traversing each element in the entire similarity matrix and extracting the elements whose element values ​​meet the preset conditions, a candidate set of elements is obtained. Each element of the candidate set reflects the similarity between the suspected trademark pattern and its associated compared trademark. Therefore, centralized operations can be performed on the suspected trademark patterns in this candidate set subsequently.

[0089] Because the compared trademarks are usually massive, the corresponding compared feature matrix also contains a massive number of image feature row vectors of the compared trademarks. In addition, because a picture to be compared may contain multiple trademark patterns, the compared feature matrix constructed from it may also contain multiple row vectors. Therefore, the final similarity matrix is ​​a matrix with multiple rows and columns. The candidate set selected from the similarity matrix is ​​also likely to contain a many-to-many relationship, that is, the candidate set can generally reflect the mapping relationship between multiple suspected trademark patterns and the similarity of multiple compared trademarks. Therefore, it is necessary to further process the candidate set so that more effective and concentrated information can be output in the end.

[0090] In order to achieve deep processing of the candidate set, a variety of methods can be used, including selection using the TopK algorithm, selection based on thresholds, etc., usually a fixed number of the most similar elements are selected for the same suspected trademark, and the corresponding compared trademarks and corresponding similarities are determined for each suspected trademark based on these elements. As for the number of the most similar elements, it can be flexibly determined by those skilled in the art. In order to present the compared trademark, the name of the compared trademark and / or the image in the picture file can be output, and the similarity can be graphically presented as needed and output to the user terminal device for display.

[0091] On the one hand, the present application constructs a feature matrix to be compared for storing vectors of image features of trademark patterns contained in product images that need to be reviewed, and on the other hand, a feature matrix to be compared is pre-constructed for storing vectors of image features of compared products. The two vectors are unified into the same dimension, and then fast and efficient matrix operations are performed based on the feature matrix to be compared and the feature matrix to be compared, so as to quickly obtain the similarity between the product pattern in the image to be compared and each compared trademark in the compared trademark library with a smaller amount of calculation and memory usage. According to the similarity, the suspected infringing trademark pattern can be further determined, and the compared trademark that matches the suspected infringing trademark pattern in the image to be compared and the corresponding similarity can be selected and associated. During the whole process, the computer operation logic is clear, the operation is efficient, the programming is convenient, and the results are obtained quickly. It is convenient to deploy in Internet platforms such as e-commerce to identify trademark infringement behaviors of massive product images, respond to infringement behaviors in a timely manner, and enhance the risk prevention and control capabilities of e-commerce platforms.

[0092] The similarity comparison implemented in this application based on matrix operations constructed based on image features does not need to rely on trademark category judgment, and can be compatible with other types of similarity judgments such as copyright and other patterns prohibited by law when necessary, further expanding the scope of discovering suspected infringements. Therefore, it is particularly suitable for e-commerce platforms.

[0093] By extension, the technical solution of this application is also applicable to trademark search and can be used to assist in the determination of similar trademarks. As long as the trademark images of a country's registered trademarks are collected to construct the compared trademark library of this application, and then the technical solution of this application is applied, the features of the compared images input by the search personnel and the compared feature matrix corresponding to the compared trademark library can be compared, and the trademarks similar to the compared images can be quickly determined, providing convenience for trademark agency, examination, and review, etc.

[0094] See also Figure 2 In order to facilitate the implementation of the specific embodiment, the step S1100 includes the following steps:

[0095] Step S1110: Obtain the image to be compared:

[0096] As mentioned above, the pictures to be compared can be obtained by providing an access interface for users to call and uploading by users; they can also be obtained by querying from the product information library of the e-commerce platform through the online service implemented according to this application. Therefore, after the platform applies the technical solution of this application, it can not only find the suspected infringing product pictures through background scanning, but also find them in time when its merchant users upload their product pictures, so as to achieve the purpose of platform risk control.

[0097] Step S1120: Call a pre-trained trademark detector to perform trademark pattern detection on the image to be compared, and obtain one or more trademark patterns contained in the image to be compared:

[0098] The trademark detector can adopt many image target detection algorithms or tools commonly used by those skilled in the art to identify the trademark patterns therein, preferably a neural network model based on deep semantic learning, such as YOLO, Faster R-CNN, and SSD (Single Shot Multibox Detector) based on VGG. With the help of such tools, one or more areas corresponding to the trademark patterns are determined in the image to be compared, and the images of each area are extracted to obtain each trademark pattern accordingly.

[0099] In this embodiment, the specific neural network model Yolo-5 is preferably used to train a trademark detector. The trademark detector is pre-trained to convergence. During its training phase, product images containing trademark patterns are fed into it as training samples. Feature extraction is performed through the model to identify one or more candidate boxes of trademark patterns. The binary classification results of whether the image content in the candidate box is a trademark pattern and the coordinate information of the product picture where the candidate box is located are output. The loss value is calculated and back-propagated within the model, and the training is continuously iterated until the model converges, and it can be put into use in the technical solution of this application. Among them, the cross entropy loss function is applied to constrain the binary classification, and the generalized overlap ratio loss function is applied to constrain the coordinate information.

[0100] The pre-trained trademark detector can take the image to be compared as input and extract the trademark patterns contained therein.

[0101] Step S1130: call the pre-trained feature extractor to extract features from each of the trademark patterns to obtain its image features. The image features of each trademark pattern are represented by a row vector. The row vectors corresponding to all trademark patterns are stored in the feature matrix to be compared:

[0102] As mentioned above, in the present application, the image feature extraction of the trademark pattern can be realized with the help of a variety of image feature extraction algorithms or tools, preferably based on a neural network model of deep semantic learning, including ordinary CNN networks suitable for extracting image features, residual networks Resnet and its subsequent evolutionary versions such as the Iresnet series, etc., as long as they are suitable for image feature extraction of the trademark pattern, the corresponding image features can be obtained.

[0103] In this embodiment, Iresnet100 network model training is preferably used to implement the feature extractor. Iresnet100 is essentially an upgraded version of Resnet and is well known to those skilled in the art. The additive angular margin loss function is applied to back-propagate the model, which can converge faster and achieve efficient representation learning effect.

[0104] After the trademark pattern is detected and cropped from the trademark detector of Yolo-5, the trademark pattern is cropped into a uniform size suitable for processing by the feature extractor, and then the feature extractor performs feature extraction on each trademark pattern to obtain the image features of each trademark pattern.

[0105] After obtaining each image feature, in order to construct the corresponding feature matrix, the feature extractor further normalizes each image feature into a 2048-dimensional row vector for output, thereby obtaining multiple row vectors corresponding to multiple trademark patterns. It should be noted that the dimension of the row vector should not constitute a limitation to the present application. In other embodiments, the dimension of the row vector can be flexibly enlarged or reduced, depending on the principle of the model adopted. As long as the processing capacity of the corresponding dimension is maintained during the processing of the present application, it will not affect the implementation of the present application.

[0106] The construction of the feature matrix to be compared is completed by adding multiple row vectors obtained by feature extraction from the image to be compared to the feature matrix to be compared. Assuming that a picture to be compared contains B identified trademark patterns, it can be understood that the feature matrix to be compared is of a scale of B*2048.

[0107] This embodiment uses two specific neural network models, Yolo-5 and Iresnet100, to implement the trademark detector and feature extractor respectively. Since these two types of models have good performance in their respective functional fields, the technical solution implemented thereby can achieve a more excellent similarity discrimination effect and higher computational efficiency.

[0108] In order to further improve the performance of the trademark detector, in the preferred embodiment, the neural network structure of the trademark detector used in step S1200 is optimized. As is known to all, Yolo-5 applies a dual-tower structure of a convolutional neural network, namely FPN+PAN, in the Neck part of its structure. In this structure, the intermediate features of the image are sequentially upsampled and downsampled at three different scales, and the corresponding output is obtained in its Prediction part. In actual measurements, when the trademark pattern is recognized according to this structure, the recognition ability for the blurred image or fine-grained image in the to-be-compared picture is still limited. Therefore, this embodiment further adds a maximum scale to the dual-tower structure, so that the dual-tower structure includes upsampling and downsampling operations at least at four scales, thereby enhancing the ability of the trademark detector to obtain image details from the to-be-compared picture, so that it can further be compatible with the recognition of blurred images or fine-grained images, and improve the recognition effect. According to the trademark detector improved by this implementation, according to actual measurement statistics, its accurate recall ability for the trademark pattern in the to-be-compared picture reaches more than 95%.

[0109] See also Figure 3 In the specific embodiment proposed for the convenience of programming implementation, the step S1300 includes the following steps:

[0110] Step S1310, obtain the transposed matrix of the compared feature matrix:

[0111] As described in the previous examples, assuming that the compared feature matrix is ​​a linear space of N*2048, it needs to be transposed to facilitate subsequent calculations. Therefore, it is converted into a transposed matrix of 2048*N here.

[0112] Step S1320: perform matrix multiplication on the feature matrix to be compared and the transposed matrix to obtain a cosine matrix:

[0113] According to the principle of linear algebra, after matrix multiplication of the feature matrix to be compared and the transposed matrix, a B*N cosine matrix can be obtained. Combined with the principle of the cosine similarity algorithm, it can be understood that the value of each element in the cosine matrix represents the similarity between the trademark pattern pointed to by its subscript b (b belongs to B) and the compared trademark pointed to by its subscript n (n belongs to N). It can be understood that the higher the similarity, the smaller the feature difference between the corresponding trademark pattern and the compared trademark, and the more similar the two are; the lower the similarity, the greater the feature difference between the corresponding trademark pattern and the compared trademark, and the less similar the two are.

[0114] Step S1330: Apply a preset normalization formula to convert the cosine matrix into a similarity matrix, so as to normalize the value of each element in the cosine matrix to the similarity of each element in the similarity matrix.

[0115] The element values ​​in the cosine matrix C need to be normalized to further improve the computational efficiency and enhance its intuitiveness. Therefore, in this step, a preset normalization formula can be applied to convert it into a similarity matrix D. The normalization formula can be flexibly determined by those skilled in the art. In this embodiment, the formula is:

[0116] D=1-(C+1) / 2

[0117] This formula represents the product of adding 1 to the elements in the cosine matrix and dividing it by 2, subtracting the product from 1, normalizing the element values ​​to the [0,1] space, and reflecting the similarity from another perspective. It should be noted that after this normalization, the higher the value of the element in the similarity matrix D, the greater the difference between the associated trademark pattern and the compared trademark, and the smaller the similarity; the lower the value of the element, the smaller the corresponding difference, and the greater the similarity. It can be understood that the subsequent calculations are not separated from this meaning.

[0118] According to this embodiment, the entire process of calculating the similarity between the trademark pattern and the compared trademark is converted into operations such as matrix multiplication and matrix summation based on each other's image features. Therefore, on the one hand, it can be easily programmed and implemented, and on the other hand, its computing efficiency is extremely high. For application scenarios deployed in e-commerce systems to serve a large number of images to be compared, its excellent results in saving computing resources and improving response efficiency are self-evident.

[0119] See also Figure 4 In order to efficiently screen out suspected infringing trademark patterns, the step S1400 includes the following steps:

[0120] Step S1410: traverse each element of the similarity matrix, add the elements whose similarity meets the preset conditions to the candidate set, determine whether the trademark pattern corresponding to the element added to the candidate set and the compared trademark have a similarity relationship, and determine the trademark pattern as a suspected trademark pattern:

[0121] In combination with the disclosure in the typical embodiment of the present application, it is known that a preset condition that is adapted to the specific meaning of the similarity of the similarity matrix can be set, and some elements in the similarity matrix can be screened out according to the preset condition to determine a candidate set L. In this embodiment, the preset condition is set to a preset threshold value to adapt to the situation in the previous embodiment that the lower the value of the element in the similarity matrix, the more similar the corresponding trademark pattern is to the compared trademark. In this way, when the similarity within the element in the similarity matrix D is lower than the preset threshold value, it means that the trademark pattern b corresponding to the element is a suspected trademark pattern and should be added to the candidate set.

[0122] In order to construct the candidate set so that the elements in the set carry sufficient information, when an element in the similarity matrix D meets the preset conditions and is selected, the coordinates {x1, y1} of the element can be determined. Since the similarity matrix D is a B*N structure, x1 belongs to B and y1 belongs to N, that is, x1 refers to a corresponding trademark pattern and y1 refers to a corresponding compared trademark. Therefore, the corresponding relationship between the coordinates of the element and its similarity value d can be constructed as the element {{x1, y1}, d} in the candidate set, expressed in the form of a pair of values.

[0123] After completing the traversal of each element in the entire similarity matrix, all elements that meet the preset conditions are added to the candidate set L. Therefore, the candidate set L actually contains the mapping relationship data including similarities between all suspected trademark patterns and all suspected compared trademarks. For each element in the candidate set L, it represents the similarity association between the suspected trademark pattern x1 and the compared trademark y1, and the similarity between them is d, where x1 is the suspected trademark pattern.

[0124] Step S1420: cluster all elements in the candidate set according to the suspected trademark pattern, and obtain mapping relationship data including corresponding similarities between each suspected trademark pattern and the compared trademark with which the suspected trademark pattern has similarity association:

[0125] In the candidate set L, for the same suspected trademark pattern x n , there may be multiple compared trademarks n In this case, these elements can be clustered according to whether they belong to the same suspected trademark pattern, thereby obtaining the mapping relationship data including the corresponding similarity between each suspected trademark pattern and the compared trademark with which it has a similarity association, that is, the pair value. The corresponding subset L can be constructed for each clustered suspected trademark pattern n , so that subsequent calculations can be performed based on these subsets.

[0126] Step S1430: Sort the compared trademarks associated with each suspected trademark pattern according to the similarity, and match the first several compared trademarks that are most similar to the suspected trademark pattern.

[0127] After constructing each subset, each element within the subset can be sorted from small to large according to the similarity d in its pair value to obtain a more readable result. In order to obtain the first several compared trademarks that are most similar to each suspected trademark pattern, each subset should theoretically contain the said number of elements, but this fixed number should be handled flexibly in actual application. It should be understood that since the compared trademarks that meet the preset conditions for the similarity corresponding to each suspected trademark pattern may not be equal, it is allowed that the elements in each of the described subsets are unequal. After all the sorted subsets are connected in sequence, a new result set can be formed. The result set contains multiple pair values. These pair values ​​are sorted from small to large according to the suspected trademark pattern as the first sorting index and the similarity therein as the second sorting index. In fact, a list is formed to complete the matching of the first several compared trademarks that are most similar to each suspected trademark pattern.

[0128] This embodiment implements the operation of the TopK algorithm in a convenient and efficient manner through set-based operations, which helps to quickly screen out data related to suspected trademark patterns.

[0129] See also Figure 5 In an embodiment further extended for visual display, step S1430 includes the following steps:

[0130] Step S1431: Based on the clustered candidate set, the compared trademarks associated with each suspected trademark pattern are sorted according to the similarity:

[0131] This step is performed by: on the basis of the principle of the previous embodiment, n The internal elements can be sorted from small to large according to the similarity of the values ​​of each element.

[0132] Step S1432: Filter out the first several compared trademarks that are most similar to the suspected trademark pattern to achieve matching, determine the index identifier of the compared trademark that matches the suspected trademark pattern and the similarity between the suspected trademark pattern and the compared trademark:

[0133] In the case where the similarity matrix is ​​the normalized similarity matrix in the above-mentioned embodiments, once the elements in each subset are determined, the matching and screening of the first several compared trademarks that are most similar to the suspected trademark pattern is actually realized, and the similarity mapping relationship data between each suspected trademark pattern and its most similar compared trademark is also stored in the corresponding subset in the form of a pair value. Therefore, the subscript y1 in the pair value {{x1,y1}, d} can actually be regarded as the identifier of the corresponding compared trademark, of course, x1 is the index identifier of the corresponding suspected trademark pattern, and d is the similarity between the two.

[0134] Step S1433: According to the index identifier, a preset index library is queried to determine the name of the compared trademark. The index library pre-stores mapping relationship data between the name of the compared trademark and the index identifier of the compared trademark in the compared feature library. The index identifier is correspondingly transmitted between the compared feature matrix, the similarity matrix, and the candidate set:

[0135] In examining the implementation process of the technical solution of the present application, it can be seen that in the process of constructing the N*2048 compared feature matrix, an index library is constructed, and a corresponding relationship is established between the name of the compared trademark and the sorting row number of the compared trademark in the compared feature matrix. Therefore, given the row number of a compared trademark in the compared feature matrix, the index library can be queried to determine its corresponding name, and this row number is essentially also an index identifier.

[0136] When the B*2048 feature matrix to be compared is multiplied by the 2048*N transposed matrix of the N*2048 feature matrix to be compared, a B*N cosine matrix is ​​obtained. At this time, the trademark pattern in the image to be compared remains unchanged by row B, while the trademark to be compared remains by column N. Therefore, the row number in the feature matrix to be compared is transferred to the column number in the cosine matrix.

[0137] Then, in the process of normalizing the cosine matrix, there is no change in the matrix structure. Therefore, the similarity matrix also inherits the corresponding relationship between the row numbers of the compared trademarks in the compared feature matrix. Specifically, the row numbers in the compared feature matrix are converted into column numbers in the similarity matrix.

[0138] Furthermore, when constructing the candidate set and the related subset, by establishing the pair values ​​in the set, the row number x1 and column number y1 in the similarity matrix are included in the pair values. It can be seen that these sets all inherit the corresponding relationship of the row numbers in the compared trademark.

[0139] According to this process, it can be seen that in the index library, the index identifier of the compared trademark is actually transmitted in each link of the entire process of the technical solution of the present application. Therefore, through y1 in each pair of values ​​of the set, the index library can be queried to obtain the name of the compared trademark corresponding to it. As mentioned above, the object corresponding to the index identifier stored in the index library can be the name of the compared trademark or various related information such as the access address of its image file. In this case, not only the name of the compared trademark can be obtained, but also the related information of the compared trademark can be obtained on demand, such as the name information, contact information, etc. of the trademark owner, so as to obtain richer output information.

[0140] Step S1434: outputting the mapping relationship data of each suspected trademark pattern and the name of the compared trademark it matches, as well as the similarity between the two:

[0141] In order to serve the needs of e-commerce platforms, it is often hoped that the result data formed by matching the technical solution of this application can provide relatively rich output information, especially how many suspected trademark patterns are extracted from the pictures to be compared, which compared trademarks have similarity associations with each suspected trademark pattern, and the specific similarities between the suspected trademark patterns and the related compared trademarks, etc. Therefore, it is appropriate to display this information in a centralized list.

[0142] In this embodiment, in order to meet the needs of the e-commerce platform and facilitate subsequent manual proofreading, for each suspected trademark pattern, the main focus is on the name of the associated compared trademark and the similarity between each compared trademark, so that the management user can see it at a glance. Since the screening and matching of these data has been completed through the above-mentioned collection, in this step, it is only necessary to combine the compared trademark name queried from the index library, format each pair of values ​​in the collection and output it as the result.

[0143] This embodiment provides a more detailed solution for visualizing the matching results, so that after the similar trademark matching operation is performed, intuitive comparison data can be obtained, which is convenient for the backend management user to further identify and further improve the accuracy of determining trademark similarity and trademark infringement.

[0144] See also Figure 6 The embodiment of the present application further provides a trademark pattern matching device, which includes a preparation module 1100 for comparison, a preparation module 1200 for comparison, a similarity calculation module 1300 and a screening matching module 1400, wherein the preparation module 1100 for comparison is used to obtain the image features of the trademark pattern contained in the image to be compared, and construct a feature matrix to be compared, wherein each row vector stores the image features of a single trademark pattern; the preparation module 1200 for comparison is used to obtain a compared feature matrix of a preset compared trademark library, wherein each row vector stores the image features of a single compared trademark in the compared trademark library, The trademark pattern and the image features of the compared trademark are both represented as row vectors with the same dimension; the similarity calculation module 1300 is used to calculate the similarity based on the feature matrix to be compared and the compared feature matrix to obtain a similarity matrix, wherein each row vector stores the similarity between a single trademark pattern and all the compared trademarks; the screening and matching module 1400 is used to screen out elements whose similarity meets preset conditions from the similarity matrix to determine the suspected trademark pattern that has a similarity association with the compared trademark from the trademark pattern, and match each suspected trademark pattern with the first several compared trademarks that are most similar to it.

[0145] In a specific embodiment, the comparison preparation module 1200 includes: a picture receiving submodule, used to obtain the picture to be compared; a trademark detection submodule, used to call a pre-trained trademark detector to perform trademark pattern detection on the picture to be compared, and obtain one or more trademark patterns contained in the picture to be compared; a feature extraction submodule, used to call a pre-trained feature extractor to perform feature extraction on each of the trademark patterns, and obtain its image features. The image feature of each trademark pattern is represented by a row vector, and the row vectors corresponding to all trademark patterns are stored in the feature matrix to be compared.

[0146] In a preferred embodiment, during the intermediate process of detecting the trademark pattern, the trademark detector applies a convolutional neural network dual-tower structure to output the final image features, and the dual-tower structure includes upsampling and downsampling operations at at least four scales.

[0147] In a preferred embodiment, the compared preparation module 1200 is configured to take the compared trademark in the compared trademark library as input, call the pre-trained feature extractor to extract the image features of the compared trademark, and normalize the image features into row vectors and add them to the compared feature matrix.

[0148] In a specific embodiment, the similarity calculation module 1300 includes: a matrix transposition submodule, used to obtain the transposed matrix of the feature matrix to be compared; a cosine calculation submodule, used to perform matrix multiplication of the feature matrix to be compared with the transposed matrix to obtain a cosine matrix; a similarity normalization submodule, used to apply a preset normalization formula to convert the cosine matrix into a similarity matrix, so as to normalize the numerical values ​​of each element in the cosine matrix to the similarity of each element in the similarity matrix.

[0149] In a specific embodiment, the screening and matching module 1400 includes: a traversal and screening submodule, which is used to traverse each element of the similarity matrix, add the elements whose similarity meets the preset conditions to the candidate set, determine that the trademark pattern corresponding to the element added to the candidate set and the compared trademark have a similarity association, and determine the trademark pattern as a suspected trademark pattern; a clustering and mapping submodule, which is used to cluster all elements in the candidate set according to the suspected trademark pattern, and obtain mapping relationship data including corresponding similarities between each suspected trademark pattern and the compared trademark with which there is a similarity association; a sorting and matching submodule, which is used to sort the compared trademarks associated with each suspected trademark pattern according to the similarity, and match the first several compared trademarks with the lowest similarity to the suspected trademark pattern.

[0150] In a further extended embodiment, the sorting and matching submodule includes: a secondary sorting submodule, which is used to sort the compared trademarks associated with each suspected trademark pattern according to similarity based on the clustered candidate set; a secondary screening submodule, which is used to screen out the first several compared trademarks that are most similar to the suspected trademark pattern to achieve matching, and determine the index identifier of the compared trademark that matches the suspected trademark pattern and the similarity between the suspected trademark pattern and the compared trademark; a secondary query submodule, which is used to query a preset index library according to the index identifier to determine the name of the compared trademark, and the index library pre-stores the mapping relationship data between the name of the compared trademark and the index identifier of the compared trademark in the compared feature library, and the index identifier is correspondingly transmitted between the compared feature matrix, the similarity matrix, and the candidate set; a secondary output submodule, which is used to output the mapping relationship data of each suspected trademark pattern and the name of the compared trademark that matches it and the similarity between the two.

[0151] The present application also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0152] like Figure 7 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor may implement a trademark pattern matching method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor may execute a trademark pattern matching method. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0153] In this embodiment, the processor is used to execute Figure 6The memory stores the program code and various data required to execute the above modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all submodules in the trademark pattern matching device, and the server can call the program code and data of the server to execute the functions of all submodules.

[0154] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the trademark pattern matching method of any of the above embodiments.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0156] It should be understood that the technical solution of the present application, in addition to being applicable to the need for e-commerce platforms to detect trademark infringement, is also applicable to trademark similarity judgments in application scenarios with the same characteristics as e-commerce titles. In this regard, those skilled in the art should be aware that the application of the technical solution of the present application to other scenarios outside the e-commerce field still falls within the scope of equivalent replacement defined in the present application.

[0157] In summary, the present application implements the calculation and judgment of trademark approximate matching based on matrix operation, with high calculation efficiency, rapid response and low overhead, and is particularly suitable for e-commerce platforms to identify suspected trademark infringement information from e-commerce product images.

[0158] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0159] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A trademark pattern matching method, characterized in that: The steps include: Obtaining image features of the trademark pattern contained in the image to be compared, and constructing a feature matrix to be compared, wherein each row vector stores the image feature of a single trademark pattern; Obtaining a compared feature matrix of a preset compared trademark library, wherein each row vector stores an image feature of a single compared trademark in the compared trademark library, and the trademark pattern and the image feature of the compared trademark are both represented as row vectors with the same dimension; Calculate similarity based on the feature matrix to be compared and the feature matrix to be compared, and obtain a similarity matrix, wherein each row vector stores the similarity between a single trademark pattern and all the compared trademarks; Selecting elements whose similarity meets preset conditions from the similarity matrix to determine suspected trademark patterns that have similarity association with the compared trademark from the trademark patterns, and matching the first several compared trademarks that are most similar to each suspected trademark pattern; The step of obtaining the image features of the trademark pattern contained in the image to be compared and constructing the feature matrix to be compared includes: Get the image to be compared; Calling a pre-trained trademark detector to perform trademark pattern detection on the image to be compared, and obtaining one or more trademark patterns contained in the image to be compared, wherein in the intermediate process of the trademark detector detecting the trademark patterns, a convolutional neural network double-tower structure is applied to output the final image features thereof, wherein the double-tower structure includes upsampling and downsampling operations at at least four scales; The pre-trained feature extractor is called to extract features from each of the trademark patterns to obtain its image features. The image features of each trademark pattern are represented by a row vector, and the row vectors corresponding to all trademark patterns are stored in a feature matrix to be compared.

2. The trademark pattern matching method according to claim 1, characterized in that: In the step of obtaining the compared feature matrix of the preset compared trademark library, the compared trademark in the compared trademark library is used as input, and the pre-trained feature extractor is called to extract the image features of the compared trademark. The image features are normalized into row vectors and added to the compared feature matrix.

3. The trademark pattern matching method according to claim 1, characterized in that: Calculating similarity based on the feature matrix to be compared and the feature matrix to be compared to obtain a similarity matrix includes the following steps: Obtain the transposed matrix of the compared feature matrix; Performing matrix multiplication of the feature matrix to be compared and the transposed matrix to obtain a cosine matrix; The cosine matrix is ​​converted into a similarity matrix using a preset normalization formula, so that the numerical value of each element in the cosine matrix is ​​normalized to the similarity of each element in the similarity matrix.

4. The trademark pattern matching method according to claim 1, characterized in that: Selecting elements whose similarity meets preset conditions from the similarity matrix to determine suspected trademark patterns that have similarity association with the compared trademark from the trademark patterns, and determining the top several compared trademarks that are most similar to each suspected trademark pattern, including the following steps: Traversing each element of the similarity matrix, adding the elements whose similarity meets the preset conditions to the candidate set, determining that the trademark pattern corresponding to the element added to the candidate set and the compared trademark have a similarity association, and determining the trademark pattern as a suspected trademark pattern; Clustering all elements in the candidate set according to the suspected trademark pattern, obtaining mapping relationship data including corresponding similarities between each suspected trademark pattern and the compared trademark associated with it; The compared trademarks associated with each suspected trademark pattern are sorted according to the similarity, and the first several compared trademarks that are most similar to the suspected trademark pattern are matched.

5. The trademark pattern matching method according to claim 4, characterized in that: Matching the first several compared trademarks that are most similar to the suspected trademark pattern includes the following steps: Based on the clustered candidate set, the compared trademarks associated with each suspected trademark pattern are sorted according to the similarity; Screen out the first several compared trademarks that are most similar to the suspected trademark pattern to achieve matching, determine the index identifier of the compared trademark that matches the suspected trademark pattern and the similarity between the suspected trademark pattern and the compared trademark; The name of the compared trademark is determined by searching a preset index library according to the index identifier, wherein the index library pre-stores mapping relationship data between the name of the compared trademark and the index identifier of the compared trademark in the compared feature library, and the index identifier is correspondingly transmitted between the compared feature matrix, the similarity matrix, and the candidate set; Output the mapping relationship data of each suspected trademark pattern and the name of the compared trademark it matches, as well as the similarity between the two.

6. A trademark pattern matching device, characterized in that: It includes: A comparison preparation module is used to obtain the image features of the trademark pattern contained in the image to be compared, and construct a comparison feature matrix, in which each row vector stores the image features of a single trademark pattern; A comparison preparation module is used to obtain a compared feature matrix of a preset compared trademark library, wherein each row vector stores the image features of a single compared trademark in the compared trademark library, and the trademark pattern and the image features of the compared trademark are both represented as row vectors with the same dimension; A similarity calculation module, used for calculating similarity based on the feature matrix to be compared and the compared feature matrix, and obtaining a similarity matrix, wherein each row vector stores the similarity between a single trademark pattern and all compared trademarks; A screening and matching module, used to screen out elements whose similarity meets preset conditions from the similarity matrix to determine suspected trademark patterns that have similarity association with the compared trademark from the trademark patterns, and match each suspected trademark pattern with the first several compared trademarks that are most similar to it; The step of obtaining the image features of the trademark pattern contained in the image to be compared and constructing the feature matrix to be compared includes: Get the image to be compared; Calling a pre-trained trademark detector to perform trademark pattern detection on the image to be compared, and obtaining one or more trademark patterns contained in the image to be compared, wherein in the intermediate process of the trademark detector detecting the trademark patterns, a convolutional neural network double-tower structure is applied to output the final image features thereof, wherein the double-tower structure includes upsampling and downsampling operations at at least four scales; The pre-trained feature extractor is called to extract features from each of the trademark patterns to obtain its image features. The image features of each trademark pattern are represented by a row vector, and the row vectors corresponding to all trademark patterns are stored in a feature matrix to be compared.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the trademark pattern matching method according to any one of claims 1 to 5.

8. A storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the trademark pattern matching method as claimed in any one of claims 1 to 5.

Images