Commodity infringement detection method and device, equipment, medium and product

Abstract

The application relates to a commodity infringement detection method and device, equipment, medium and product. The method comprises the following steps: obtaining a commodity picture and text information in commodity information of a target commodity, wherein the text information comprises commodity description information of the target commodity; extracting a trademark image set from the commodity picture, wherein the trademark image set comprises one or more trademark images; determining an infringement probability of each trademark image according to the trademark image set and the text information; and determining a trademark image suspected of infringing in the trademark image set according to the infringement probability. Since the text information is composed of the commodity description information of the target commodity, the commodity description information can provide reference information in terms of word semantics for identification of the content of the commodity picture, so that whether the trademark image is suspected of infringing can be determined more effectively, and it is ensured that whether the commodity picture of the target commodity contains an unauthorized trademark can be accurately determined.

Description

Technical Field

[0001] This application relates to the field of e-commerce information technology, and in particular to a method for detecting product infringement and the corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Intellectual property is a widely accepted international legal system, and trademark rights are one type of intellectual property. In the wave of internet globalization, e-commerce platforms have a legal and ethical obligation to protect the legitimate use of trademark rights. Therefore, they need to detect information suspected of trademark infringement in order to prevent the malicious use and plagiarism of legitimate trademarks and effectively maintain a healthy market order.

[0003] With the rapid growth of transaction volume on e-commerce platforms, manual review of images uploaded by merchants is too inefficient and impractical, thus necessitating the use of technical detection methods. Existing methods for detecting suspected trademark infringement typically rely solely on product images for identification, often resulting in inaccurate detection. For example, a merchant may intend to showcase clothing worn by a model that does not infringe on trademarks, but the shoelaces worn by the model may bear trademarks and be identified as infringing because they are in a trademark database, leading to the merchant being mistakenly identified as an infringer. Similarly, an electronic accessory manufacturer may intend to demonstrate compatibility with certain brands of equipment, but because these brands of equipment bear trademarks from other companies, the product may be judged as infringing on these unauthorized brands, leading to the merchant being mistakenly identified as a suspected infringer and subject to strict risk control.

[0004] This shows that traditional trademark detection methods are relatively crude in determining whether a trademark is infringing, and are prone to misjudgment. Related detection technologies need to be further improved. Summary of the Invention

[0005] The purpose of this application is to solve the above-mentioned problems by providing a method for detecting product infringement and corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product.

[0006] To suit the various purposes of this application, the following technical solution is adopted:

[0007] In one respect, a method for detecting product infringement is provided to suit one of the purposes of this application, comprising the following steps:

[0008] Obtain product images and text information from the product information of the target product, wherein the text information includes the product description information of the target product;

[0009] Extract a set of trademark images from the product images, the set of trademark images containing one or more trademark images;

[0010] The probability of infringement for each trademark image is determined by combining the trademark image set with the text information.

[0011] The trademark images suspected of infringement in the trademark image set are determined based on the infringement probability.

[0012] Optionally, determining the infringement probability of each trademark image based on the trademark image set and the text information includes the following steps:

[0013] Extract the image features of each trademark image in the trademark image set;

[0014] Extract the text features of the text information;

[0015] The text features and the image features are fused to obtain fused features;

[0016] The infringement probability corresponding to each trademark image is obtained by normalizing the interaction features between the fusion features and the text features.

[0017] Optionally, the text features and the image features are fused to obtain fused features, including the following steps:

[0018] An attention layer is applied to perform initial feature fusion with the text features and image features to obtain initial features;

[0019] The image features are subjected to multiple nonlinear activations to obtain corrected features;

[0020] The initial features and the modified features are fused together to obtain the fused features.

[0021] Optionally, an attention layer is applied to perform initial feature fusion with the text features and image features to obtain initial features, including the following steps:

[0022] The image features are used as query vectors, and the text features are used as key and value vectors, which are then input into the attention layer.

[0023] The attention layer interacts with and normalizes the query vector and the key vector to obtain a weight matrix;

[0024] The attention layer matches the value vector with the weight matrix to obtain initial features.

[0025] Optionally, the image features are subjected to multiple nonlinear activations to obtain corrected features, including the following steps:

[0026] The image features are fully connected and then subjected to a first nonlinear activation to obtain the initial corrected features.

[0027] The initial modified features are fully connected and then subjected to a second nonlinear activation to obtain the final modified features.

[0028] Optionally, normalization is performed based on the interaction features between the fused features and the text features to obtain the infringement probability corresponding to each trademark image, including the following steps:

[0029] The fused features are combined with the text features through shallow feature interaction to obtain shallow interaction features;

[0030] An attention layer is applied to perform deep feature interaction on the shallow interaction features to obtain deep interaction features;

[0031] The deep interactive features are transformed and pooled into high-dimensional vectors, with each dimension corresponding to a trademark image;

[0032] The probability of infringement corresponding to each trademark image is determined based on this high-dimensional vector.

[0033] Optionally, determining the trademark images suspected of infringement in the trademark image set based on the infringement probability includes the following steps:

[0034] The image features of each trademark image in the trademark image set are classified and mapped to determine the legal trademarks corresponding to each trademark image;

[0035] The trademark images suspected of infringement in the trademark image set are selected based on the corresponding infringement probability of each trademark image;

[0036] Output the mapping relationship data between the allegedly infringing trademark image and its corresponding legitimate trademark.

[0037] On the other hand, to meet one of the purposes of this application, a product infringement detection device is provided, comprising an image and text acquisition module, a trademark extraction module, a joint analysis module, and an infringement determination module, wherein: the image and text acquisition module is used to acquire product images and text information from the product information of a target product, the text information including product description information of the target product; the trademark extraction module is used to extract a set of trademark images from the product images, the set of trademark images containing one or more trademark images; the joint analysis module is used to jointly determine the infringement probability of each trademark image based on the set of trademark images and the text information; and the infringement determination module is used to determine the trademark images suspected of infringement in the set of trademark images based on the infringement probability.

[0038] Optionally, the joint analysis module includes: a trademark feature extraction submodule, used to extract image features of each trademark image in the trademark image set; a text feature extraction submodule, used to extract text features of the text information; a multi-modal feature fusion submodule, used to fuse the text features with the image features to obtain fused features; and an infringement probability determination submodule, used to normalize based on the interaction features of the fused features and the text features to obtain the infringement probability corresponding to each trademark image.

[0039] Optionally, the multi-modal feature fusion submodule includes: an initial fusion unit, used to apply an attention layer to perform initial feature fusion of the text features and image features to obtain initial features; an activation correction unit, used to perform multiple nonlinear activations on the image features to obtain corrected features; and a secondary fusion unit, used to perform secondary feature fusion of the initial features and the corrected features to obtain fused features.

[0040] Optionally, the initial fusion unit includes: a vector input subunit, used to input the image features as a query vector and the text features as a key vector and a value vector into the attention layer; a weight extraction subunit, used by the attention layer to interact with and normalize the query vector and the key vector to obtain a weight matrix; and a feature generation subunit, used by the attention layer to match the value vector with the weight matrix to obtain initial features.

[0041] Optionally, the activation correction unit includes: an initial correction subunit, used to perform a first nonlinear activation after fully connecting the image features to obtain initial corrected features; and a secondary correction subunit, used to perform a second nonlinear activation after fully connecting the initial corrected features to obtain final corrected features.

[0042] Optionally, the infringement probability determination submodule includes: a shallow interaction unit, used to perform shallow feature interaction between the fused features and the text features to obtain shallow interaction features; a deep interaction unit, used to apply an attention layer to perform deep feature interaction on the shallow interaction features to obtain deep interaction features; a vector transformation unit, used to transform and pool the deep interaction features into a high-dimensional vector, such that each dimension corresponds to a trademark image; and a probability calculation unit, used to calculate and determine the infringement probability corresponding to each trademark image based on the high-dimensional vector.

[0043] Optionally, the infringement determination module includes: a trademark mapping submodule, used to classify and map the image features of each trademark image in the trademark image set to determine the corresponding legitimate trademark; an image determination submodule, used to select the suspected infringing trademark images in the trademark image set according to the infringement probability corresponding to each trademark image; and a data output submodule, used to output the mapping relationship data between the suspected infringing trademark images and their corresponding legitimate trademarks.

[0044] In another aspect, a computer device provided for one of the purposes of this application includes a central processing unit and a memory, the central processing unit being used to invoke and run a computer program stored in the memory to perform the steps of the commodity infringement detection method described in this application.

[0045] In another aspect, a computer-readable storage medium is provided to suit another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the described product infringement detection method, which, when invoked by a computer, performs the steps included in the method.

[0046] In another aspect, a computer program product provided for another purpose of this application includes a computer program / instructions that, when executed by a processor, implement the steps of the commodity infringement detection method described in any embodiment of this application.

[0047] Compared to existing technologies, this application has several advantages, including at least the following: When identifying trademark infringement in product images of target goods, this application acquires information from two modalities. One modal is a set of trademark images consisting of one or more trademark images in the product image of the target goods, and the other is the text information of the target goods. Then, based on the information from both modalities, the infringement probability of each trademark image in the set mapped to each legitimate trademark is determined. Based on the infringement probability, the trademark images suspected of infringement in the set are identified, thus enabling the detection of whether trademark images in product images constitute infringement. Since the text information is composed of the product description information of the target goods, the product description information can provide semantic reference information for the identification of the content of the product image. Therefore, in the process of trademark image infringement detection, combining the trademark image with the text information of the target goods can more effectively determine whether the trademark image is suspected of infringement, making the identification of suspected infringing trademark images more accurate, thereby ensuring that the product image of the target goods contains unauthorized trademarks accurately. Attached Figure Description

[0048] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0049] Figure 1 This is a flowchart illustrating a typical embodiment of the product infringement detection method of this application.

[0050] Figure 2 This is a schematic diagram of the network structure of an exemplary trademark recognition model in this application.

[0051] Figure 3 This is a flowchart illustrating the process of determining the probability of infringement based on the product identification model in an embodiment of this application.

[0052] Figure 4 This is a flowchart illustrating the process of feature fusion of text features and image features in an embodiment of this application.

[0053] Figure 5 This is a schematic diagram of the structure of the feature fusion network within the trademark recognition model used to perform the feature interaction between the text features and the image features, as described in an embodiment of this application.

[0054] Figure 6 This is a flowchart illustrating the process of obtaining initial features using an attention layer in an embodiment of this application.

[0055] Figure 7 This is a schematic diagram of the structure of the attention layer used to obtain initial features within the trademark recognition model, as described in an embodiment of this application.

[0056] Figure 8 This is a flowchart illustrating the process of activating image features twice, as described in an embodiment of this application.

[0057] Figure 9 This is a schematic diagram of the squeeze door structure within the trademark recognition model used to achieve nonlinear activation of image features, as described in an embodiment of this application.

[0058] Figure 10 This is a flowchart illustrating the process of determining deep interaction features and calculating the probability of infringement based on fusion features and text features in an embodiment of this application.

[0059] Figure 11 This is a schematic diagram of the structure of the transformation network used to perform the infringement probability calculation process within the trademark recognition model, as described in an embodiment of this application.

[0060] Figure 12 This is a flowchart illustrating the process of selecting the suspected infringing trademark image based on the probability of infringement in an embodiment of this application.

[0061] Figure 13This is a schematic diagram of the product infringement detection device of this application;

[0062] Figure 14 This is a schematic diagram of the structure of a computer device used in this application. Detailed Implementation

[0063] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0064] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0065] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0066] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0067] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0068] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0069] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client for access.

[0070] Unless otherwise specified, the neural network models referenced or potentially referenced in this application may be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly. In some embodiments, when running on the client, the corresponding intelligence may be acquired through transfer learning in order to reduce the requirements on the client's hardware resources and avoid excessive consumption of the client's hardware resources.

[0071] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0072] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0073] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0074] The product infringement detection method of this application can be programmed into a computer program product and deployed on a client or server for execution. For example, in an exemplary application scenario of this application, it can be deployed on the server of an e-commerce platform. In this way, the method can be executed by human-computer interaction with the process of the computer program product through a graphical user interface by accessing the interface opened after the computer program product is run.

[0075] Please see Figure 1 The product infringement detection method of this application, in its typical embodiment, includes the following steps:

[0076] Step S1100: Obtain the product image and text information from the product information of the target product, wherein the text information includes the product description information of the target product;

[0077] In e-commerce platform applications, each product is processed as a relatively independent unit of information, published and updated by the merchant user of the online store on the e-commerce platform, and made available for consumer users to browse and place orders. The online store can be an independent website, which independently maintains its own product database and can identify whether product images contain trademark infringement information by installing the computer program product obtained according to this application. Each product is described with its corresponding product information, which typically includes two main categories: product images and text information.

[0078] The product images mentioned above are typically used to display graphics of the corresponding product or its related products, or their usage status. For example, when the product is a garment, one of its product images can be used to show the effect of a model wearing the garment; similarly, when a product needs to display graphics of its complementary products, one of its product images can be used to show the effect of including the complementary products. That is, the product images are allowed to contain content other than the graphic of the product itself, and this content may also include other products besides the current product.

[0079] The aforementioned text information refers broadly to all product description information associated with the product storage and suitable for provision in text form, including but not limited to any one or more of the product title, product attribute data, product detail text, and product profile tags. In terms of usage, the product description information is generally used to describe any specific information of the product, such as its title, brand, name, model, category, purpose, function, nature, characteristics, and usage.

[0080] In one embodiment, the product title and its details text can be concatenated together from the product database of an online store to form its text information. In another embodiment, keywords from the product title and details text, as well as product attribute data and product profile tags, can be further obtained and merged into a word segmentation set for use as the text information.

[0081] When an online store merchant needs to publish a target product, they enter the corresponding product information on the product publishing page and then submit it to the backend so that the product information can be stored in the product database.

[0082] When it is necessary to identify a specific product in the product database, or a specific product posted by a merchant user in real time, the product information of the target product can be obtained, and a single product image and text information can be extracted from it as joint input to start the trademark infringement identification process.

[0083] If the product information of a target product contains multiple product images, the infringement identification can be performed on each product image individually, in conjunction with the aforementioned text information.

[0084] Step S1200: Extract a trademark image set from the product image, wherein the trademark image set contains one or more trademark images;

[0085] Regardless of the content of the product images, any content related to the product or goods may carry its own trademark image. For product images used for trademark infringement identification, each trademark image can be extracted to form a trademark image set A. TM .

[0086] The detection of trademark images in the product images can be performed using a pre-trained object detection model that has reached a convergent state. This object detection model is generally implemented using deep learning-based models, such as the R-CNN series, the YOLO series, and the SSD (Single Shot MultiBox Detector) series. The R-CNN series is a representative algorithm based on region detection, YOLO is a representative algorithm based on region extraction, and SSD is an algorithm improved upon the first two series.

[0087] The R-CNN series typically includes different specific models such as R-CNN, SPPNet, FastR-CNN, and FasterR-CNN, while the YOLO series also has multiple versions available. These object detection models are all suitable for identifying target image regions from a given image, thereby obtaining the corresponding target image based on the target image region.

[0088] In one embodiment, YOLOv5 can be used as the object detection model, connected to a classifier, and fine-tuned using a sufficient amount of training samples. The training samples are product images, which include one or more trademark images. Each training sample provides corresponding training labels for each trademark image to supervise the model training and enable it to learn the ability to accurately identify one or more trademark image regions from a given product image.

[0089] Therefore, after preprocessing the product image of the target product according to the specifications of the input image using the target detection model, the target detection model is used to perform target detection on the preprocessed product image. This allows the identification of the trademark image regions corresponding to the trademarks carried by each item in the product image, and outputs the coordinate information of each trademark image region. Furthermore, based on the coordinate information output by the target detection model, the trademark images corresponding to the coordinate information of each trademark image region are cropped from the product image, thus obtaining the trademark image set A corresponding to the product image. TM Depending on the actual situation of the trademark images carried in the product images, the set of trademark images may include one or more trademark images. Of course, if no trademark image is detected in the product images, no further processing of this application is necessary.

[0090] like Figure 2 In the trademark recognition model shown, the target detection model is used as a front-end component to obtain one or more trademark images from a product image, forming a trademark image set A. TM The obtained set of trademark images is used as input to the image feature extraction model.

[0091] Step S1300: Determine the infringement probability of each trademark image based on the trademark image set and the text information;

[0092] The trademark image set obtained from the product image of the target product and the text information of the target product can be used as two modalities of information to jointly determine the infringement probability corresponding to each trademark image in the trademark image set.

[0093] In one embodiment, on the one hand, the trademark image set A can be targeted separately. TM Each trademark image in the dataset has its corresponding image features extracted in order to obtain the image features corresponding to the entire trademark image set. im On the other hand, the corresponding text features can be obtained from the text information. text The image features of each trademark image are featured. imWith the text feature text All features are unified into vectors of the same dimension. Then, the image features and text features are combined into joint feature information. im_sq_sa The input is fed into a preset trademark recognition model to determine the confidence level P corresponding to each dimension. Each confidence level can be used as the infringement probability of the trademark image in its corresponding dimension.

[0094] The trademark recognition model can be pre-constructed and pre-trained to a convergent state, enabling it to learn the probability of infringement of each trademark image in the given trademark image set corresponding to a legitimate trademark in a pre-defined trademark database, based on a given set of trademark images and text information. For example... Figure 2 A principle-based example is provided, in which the trademark recognition model employs an image feature extraction model to realize the image feature features. im The text features are obtained by using a text feature extraction model. text The image features are extracted, and then a feature fusion network is used to fuse the image features. im With the text feature text Comprehensive acquisition of joint feature information feat im_sq_sa Afterwards, the joint feature information features im_sq_sa It is in the image feature feat im Based on the above, the text features are integrated. text The semantic information results can be transformed into a vector with the same dimensions as the number of trademark images in the trademark image set through a transformation network. Then, the confidence level P corresponding to each dimension, i.e. each trademark image, is calculated based on the vector to determine the infringement probability.

[0095] It is easy to understand that the joint feature information not only includes the image features of each trademark image itself, but also incorporates the text features of the target product. These text features enhance the semantics of the image features, providing important reference information for calculating the infringement probability using the joint feature information. Since the source of the text features—that is, the text information—usually describes information about the target product itself and descriptive information about the contents of related product images—the semantics of this descriptive information can effectively enhance the semantics of the image features of each trademark image. The infringement probability determined based on this semantics is more accurate because it associates information from two modalities, and can distinguish whether the trademark image in the product image is a fair use of a legitimate trademark or an unauthorized use of a legitimate trademark, thereby avoiding misjudgments of fair use of trademarks.

[0096] Step S1400: Determine the trademark images suspected of infringement in the trademark image set based on the infringement probability.

[0097] According to the trademark image set A TM Corresponding image features im It can be further through Figure 2 The classification network shown performs classification mapping to determine the legitimate trademarks corresponding to the image features of each trademark image within it. Specifically, the trademark image set A... TM The corresponding image features are extracted. im Subsequently, classification mapping can be performed by connecting to a pre-trained classification network that has reached convergence. The classification space of the classification network is set with a corresponding number of categories according to the number of legal trademarks in the preset trademark database and trained until convergence. This enables the network to learn the ability to determine the legal trademarks corresponding to each trademark image in the given trademark image set based on the image features corresponding to each trademark image, thereby obtaining a set of suspected infringing trademarks A. sus The suspected infringing trademark collection A sus In this context, data can be used to represent the mapping relationship between each trademark image and its corresponding legal trademark.

[0098] At this point, the aforementioned collection of suspected infringing trademarks, A, can be used as a reference. sus The infringement probability of each trademark image in the dataset is determined. When the infringement probability reaches a preset condition, the corresponding trademark image is deemed a suspected infringing trademark image that is highly likely to infringe upon its corresponding legitimate trademark. The preset condition can be implemented according to actual needs, such as by truncating the infringement probability based on sorting or by filtering the infringement probability based on a preset threshold. The number of truncated preferred images and the preset threshold can be flexibly determined.

[0099] It should be noted that the term "suspected infringement" in this application refers to a high probability, not to an infringement confirmed by law. Therefore, in one embodiment, the suspected infringing trademark images can be further compiled into an infringing trademark set for further manual verification.

[0100] As can be seen from the above embodiments, this application has multiple advantages, including at least the following aspects: When identifying trademark infringement in product images of target goods, this application acquires information in two modalities. One modal is a set of trademark images consisting of one or more trademark images in the product images of the target goods, and the other modal is the text information of the target goods. Then, based on the information in both modalities, the probability of each trademark image in the set being mapped to a legitimate trademark is jointly determined. Based on the infringement probability, the trademark images suspected of infringement in the set are determined, thereby realizing the detection of whether the trademark images in the product images constitute infringement. Since the text information is composed of the product description information of the target goods, the product description information can provide semantic reference information for the identification of the content of the product images. Therefore, in the process of trademark image infringement detection, combining the trademark images with the text information of the target goods can more effectively determine whether the trademark images are suspected of infringement, making the identification of suspected infringing trademark images more accurate, thereby ensuring that the product images of the target goods contain unauthorized trademarks accurately.

[0101] Please see Figure 3 Based on any of the above embodiments, step S1300, determining the infringement probability of each trademark image in conjunction with the trademark image set and the text information, includes the following steps:

[0102] Step S1310: Extract the image features of each trademark image in the trademark image set;

[0103] As mentioned above, still refer to Figure 2 As shown, the trademark image set A TM Each trademark image in the image set can be individually input into an image feature extraction model to extract its features, which are then concatenated to form the image features corresponding to the trademark image set. This image feature extraction model can be a basic model based on CNNs, such as ResNet, or it can be input into an image feature extraction model suitable for batch feature extraction to directly obtain the image features corresponding to the trademark image set. This type of image feature extraction model can be a Transformer series model, such as Swin-Transformer. In summary, after feature extraction from the trademark image set using the image feature extraction model, the trademark image set A can be obtained. TM The corresponding image features are denoted as feat in this application. im It's not hard to understand that if there are B trademark images in the trademark image set, and each trademark image has a dimension of N, then the obtained image features are... im Let be the characteristic matrix of B*N.

[0104] It should be noted that the trademark images in the trademark image set may vary in size. To facilitate the standardization process of the image feature extraction model, scaling, cropping, and other methods can be used to preprocess each trademark image to obtain standardized trademark images.

[0105] Step S1320: Extract the text features of the text information;

[0106] The text information can be processed according to its specific information type. For each specific information type, word segmentation and word embedding are performed first to obtain the corresponding embedding vectors. Then, these embedding vectors are constructed into a matrix and input into a text feature extraction model for feature extraction, thereby obtaining the corresponding text features. text Suppose the text information contains T types of specific information, such as product titles and product detail text. In this case, for consistency, the dimension of the text features is still set to N. Therefore, the final text features corresponding to the text information are... text The feature matrix is ​​T*N. This achieves a unification of text features and image features in terms of vector dimension.

[0107] The text feature extraction model described herein can be implemented using an RNN-based neural network model, such as mature models like LSTM or BERT. Those skilled in the art can flexibly select the appropriate model based on the principles disclosed herein.

[0108] Step S1330: Perform feature fusion between the text features and the image features to obtain fused features;

[0109] Since the vector dimensions of image and text features have been unified during the acquisition process, in one embodiment, a feature fusion network can be used to integrate image and text features through matrix multiplication to obtain joint feature information. This joint feature information is then used as the fusion feature required for subsequent processing, denoted as feature. im_sq_sa Its matrix structure remains B*N.

[0110] In another embodiment, a feature fusion network can be implemented using weight extraction. This network performs deeper feature interaction between the image features and text features to obtain joint feature information that combines the semantics of both, which serves as the fusion feature required for subsequent processing. im_sq_sa Similarly, its matrix structure is still B*N.

[0111] In summary, by interacting the image features with the text features, a fusion feature that integrates the semantics of both can be obtained. This enables the semantic integration of the trademark image in the product image of the target product and the text information of the target product, so that the obtained fusion feature can effectively represent the correlation information between the trademark image and the text information.

[0112] Step S1340: Normalize the interaction features between the fused features and the text features to obtain the infringement probability corresponding to each trademark image.

[0113] Furthermore, the fused features are processed through a transformation network. im_sq_sa After further feature interaction with the text features, specific feature interaction methods may include matrix multiplication to obtain a B*T matrix, thereby obtaining intermediate features that enhance the semantic information of the text features. The number of trademark images corresponding to the intermediate features is further transformed into B and normalized to obtain a B*1 vector. At this point, the transformation network can calculate the confidence level corresponding to each dimension based on the vector. Each confidence level corresponds to a specific trademark image because it corresponds to a dimension, thus obtaining the confidence level corresponding to each trademark image, which can be used as an indication of the probability of infringement of whether the trademark image constitutes infringement.

[0114] As can be seen from the above embodiments, by fusing the image features and text features of the target product, and further combining the text features on the basis of the fused features, the infringement probability corresponding to each trademark pattern is finally calculated. In this process, a deep interaction between the image and text modal information of the trademark image and the text information of the target product is realized. Since the semantics of the text information are deeply combined on the basis of the semantics of the trademark image, the infringement probability calculated accordingly can effectively avoid misjudgment of whether the trademark image is suspected of infringement under the guidance of the semantics of the text information, and greatly improve the accuracy of infringement detection of product images.

[0115] Please see Figure 4 Based on any of the above embodiments, step S1330, fusing the text features and the image features to obtain fused features, includes the following steps:

[0116] Step S1331: Apply an attention layer to perform initial feature fusion of the text features and image features to obtain initial features;

[0117] In order to obtain the fusion feature of the text features and the image features, im_sq_sa ,like Figure 5 As shown, the feature fusion network provides an attention layer and a squeeze gate to process the text features and image features.

[0118] First, the text features and image features are input in parallel into the attention layer. The attention layer's attention mechanism then performs feature interaction between the text and image features to obtain initial features (SA). The attention layer determines the weight information of the image features based on the text features, highlighting salient information. Finally, the weight information is fused with the text features to obtain the initial features. The initial features (SA) and their corresponding image features are then compared. im They have the same scale and are both B*N feature matrices.

[0119] Step S1332: Perform multiple nonlinear activations on the image features to obtain corrected features;

[0120] The aforementioned squeezing gate can be implemented based on a gate mechanism, whereby the image features are sequentially passed through two activation layers undergoing different types of nonlinear activation to obtain modified features. im_sq It can be understood that the scale of the modified features is also B*N. Therefore, the introduction of the gating mechanism can avoid gradient vanishing, making the model easier to train to a convergent state, and can improve the overall recognition accuracy of the trademark recognition model of this application.

[0121] Step S1333: Perform secondary feature fusion on the initial feature and the modified feature to obtain the fused feature.

[0122] Obtain the initial feature SA and the modified feature feat im_sq Then, the dot product matrix of the initial feature and the modified feature is calculated using a dot product operation. This dot product matrix can then be used as the fused feature obtained after fusing text features and image features. im_sq_sa That is, the joint feature information mentioned above.

[0123] As can be seen from the process of the above embodiments, the initial features obtained after the text features and image features interact with each other through the attention layer carry the weight information of the image features determined based on the text features and the semantic information provided by the text features themselves. The image features then undergo multiple nonlinear activations to obtain their modified features. On this basis, the initial features and modified features are fused through dot product operations. The resulting fused features achieve deep interaction between image features and text features, making the two deeply correlated while retaining effective semantic information, thus ensuring that the probability of infringement can be accurately determined.

[0124] Based on any of the above embodiments, please refer to Figure 6 Step S1331, applying an attention layer to perform initial feature fusion of the text features and image features to obtain initial features, includes the following steps:

[0125] Step S2100: Input the image features as query vectors and the text features as key vectors and value vectors into the attention layer;

[0126] like Figure 7 As shown, this application introduces an attention layer in the fusion feature network to focus on the text features. text and the image features im For the initial feature fusion, the image features are used as the query vector in the attention layer, and the text features are used as both the key and value vectors in the attention layer, so that they can be input into the attention layer and feature interaction can be performed according to the inherent Q, K, V operation mechanism of the attention layer.

[0127] Step S2200: The attention layer interacts with and normalizes the query vector and the key vector to obtain a weight matrix;

[0128] like Figure 7 As shown, in the attention layer, the image features feature im As the query vector, it matches its corresponding weight W. Q The first feature matrix is ​​obtained, and the text features are... text Match its corresponding weight W K The second feature matrix is ​​obtained. Then, the transposes of the first and second feature matrices are multiplied by a dot product to obtain a product matrix that realizes the feature interaction between the image features and the text features. This product matrix has a B*T scale. After activating and outputting it using the Softmax function, the resulting weight matrix is ​​the semantic information after the deep interaction between the semantic information of the image features and the text features. Essentially, it is also the result of highlighting the salient features in the image features based on the text features.

[0129] Step S2300: The attention layer matches the value vector with the weight matrix to obtain initial features.

[0130] Continue reading Figure 7 In the attention layer, the weight matrix with a scale of B*T, which is activated by the Softmax function, is then matched with the corresponding weights W. V The value vector, which is the text feature with a scale of T*N, is then multiplied to obtain a product matrix with a scale of B*N. This is the initial feature SA obtained after the initial feature interaction between the text feature and the image feature.

[0131] The query vector W in this embodiment Q Key vector WK Value vector W V All of these are learnable weights, which are acquired during the training of the trademark recognition model and applied in the model inference stage.

[0132] Based on the above embodiments, it is easy to understand that the initial features are obtained by multiplying the text features on the weight matrix obtained after the interaction of text features and image features. Therefore, the text features are deeply integrated again corresponding to the number B of trademark images. Thus, a preliminary comprehensive representation of the text features and the image features is achieved. Effective feature representation is the basis for guiding the trademark recognition model to accurately calculate the confidence level. It is helpful for promoting the model to train to convergence quickly and improving the model recognition efficiency.

[0133] Please see Figure 8 Based on any of the above embodiments, step S1332, which involves performing multiple nonlinear activations on the image features to obtain corrected features, includes the following steps:

[0134] Step S3100: After fully connecting the image features, perform the first nonlinear activation to obtain the initial corrected features;

[0135] Please combine Figure 9 As can be seen from the network structure of the squeeze door implemented based on the gate mechanism, the image feature features im After being input into the squeeze gate, the weights W obtained during training are adapted by the first fully connected layer therein. g1 After full connection, the input is fed into the first activation layer, and the first activation method is applied for the first nonlinear activation. The first activation method can be implemented using ReLU, thereby obtaining the initial modified features.

[0136] Step S3200: After fully connecting the initial modified features, perform a second nonlinear activation to obtain the final modified features.

[0137] The initially corrected features are further matched with their weights W by a second fully connected layer. g2 After full connection, the input is fed into the second activation layer, where a second nonlinear activation method, such as the Tanh method, is applied to obtain the final modified feature. im_sq It's not hard to understand that modifying features... im_sq It is a feature matrix with a scale of B*N.

[0138] As can be seen above, the image features are processed through two fully connected layers, each of which is modified by different nonlinear activation methods, so that the features of each trademark image can be fused and gradient vanishing can be avoided. The final modified features obtained can effectively represent the features of each trademark image and its association with other trademark images, and can also make the model easier to train.

[0139] Please see Figure 10 Based on any of the above embodiments, step S1340, which involves normalizing the interaction features of the fused features and the text features to obtain the infringement probability corresponding to each trademark image, includes the following steps:

[0140] Step S1341: Perform shallow feature interaction between the fused features and the text features to obtain shallow interaction features;

[0141] Depend on Figure 2 As can be seen from the structure of the trademark recognition model, the fused feature obtained after feature interaction between the text features and image features is... im_sq_sa It is further processed by a conversion network to calculate the probability of infringement for each trademark image.

[0142] like Figure 11 As shown, in the transformation network, the fusion feature is... im_sq_sa With the text feature text Performing a dot product operation yields a feature matrix with a scale of B*T, which fuses the features of the two entities and can serve as a shallow interaction feature between them.

[0143] Step S1342: Apply the attention layer to perform deep feature interaction on the shallow interaction features to obtain deep interaction features;

[0144] Furthermore, the shallow interaction features are input into an attention layer (MSA) to perform deep feature interaction on the vectors within it, enabling deeper feature fusion between the vectors corresponding to each trademark image, thereby obtaining the corresponding deep interaction features. The attention layer used here can be a multi-head self-attention layer, where the shallow interaction features simultaneously serve as the query vector, key vector, and value vector in the self-attention layer, performing Q, K, V operations to obtain the deep interaction features.

[0145] Step S1343: Convert and pool the deep interaction features into a high-dimensional vector, so that each dimension corresponds to a trademark image;

[0146] Furthermore, after the deep interaction features are fully connected through a fully connected layer (fc), a pooling layer (Max_pooling) is applied to perform maximum pooling operation on them, reducing their dimension to a high-dimensional vector of B*1. It is easy to understand that in this high-dimensional vector, the feature value of each dimension corresponds to a trademark image.

[0147] Step S1344: Calculate and determine the infringement probability corresponding to each trademark image based on the high-dimensional vector.

[0148] Based on the high-dimensional vector, a normalization layer (sigmoid) constructed by the sigmoid function is applied to calculate the confidence level P corresponding to each dimension. The confidence level corresponding to each dimension is the probability of infringement of the corresponding trademark image, which represents the likelihood that the corresponding trademark image contains the allegedly infringing trademark.

[0149] As can be easily understood from the above embodiments, after the transformation network performs feature interaction on the fused features and text features, the obtained deep interactive features are the result of deep fusion of the deep semantic information of the trademark image and text information of the target product. The confidence level calculated based on this result can more accurately represent the probability of each trademark image infringing on the legitimate trademarks in the trademark database.

[0150] Please see Figure 12 Based on any of the above embodiments, step S1400, determining the trademark images suspected of infringement in the trademark image set according to the infringement probability, includes the following steps:

[0151] Step S1410: Classify and map the image features of each trademark image in the trademark image set to determine the legal trademark corresponding to each trademark image;

[0152] like Figure 2 As shown, the image features corresponding to the trademark image set obtained by the image feature extraction model are... im The data is further fed into a pre-trained, converged classification network for classification mapping, obtaining the corresponding label of each trademark image to a legitimate trademark in a pre-defined trademark database. This yields the mapping relationship data between trademark images and their corresponding legitimate trademarks, i.e., the set A of suspected infringing trademarks. sus Based on the mapping data in this set of suspected infringing trademarks, the legitimate trademark corresponding to any given trademark image can be determined.

[0153] The classification network can be placed in the trademark recognition model for joint training. Its classification space can contain multiple categories corresponding to the number of legal trademarks in the trademark database. When performing classification mapping, the image features of each trademark image are mapped to each category in the classification space. The legal trademark with the highest classification probability is the legal trademark corresponding to the trademark image.

[0154] Step S1420: Select the trademark images suspected of infringement from the trademark image set according to the corresponding infringement probabilities of each trademark image;

[0155] In order to identify the trademark images suspected of infringement from the set of suspected infringing trademarks, a subset of the suspected infringing trademark images can be selected from the set of suspected infringing trademarks based on the confidence level obtained for each trademark image, that is, its probability of infringement.

[0156] In one embodiment, the set A of suspected infringing trademarks is analyzed based on the infringement probability. sus The mapping relationship data in the dataset is sorted in reverse order, so that the trademark images are sorted from highest to lowest probability of infringement. Then, based on actual needs, the mapping relationship data corresponding to the top N trademark images are selected to form a set A of suspected infringing trademark images. infri The value of N can be any positive integer other than zero. When it is 1, it indicates that the mapping relationship data corresponding to the trademark image with the highest probability of infringement is selected, that is, a single suspected infringing trademark image is identified. Of course, the value of N can also be set as needed according to the amount of content contained in the product image.

[0157] In another embodiment, a preset threshold is used in conjunction with the set of suspected infringing trademarks A. sus The infringement probabilities corresponding to each trademark image are compared. The preset threshold is an empirical threshold or a measured threshold. When a trademark image exceeds the preset threshold, its corresponding trademark image can be determined as a suspected infringing trademark image, and its mapping relationship data with the legitimate trademarks is added to the suspected infringing trademark image set A. infri Based on this, the image set A of the allegedly infringing trademarks is constructed. infri They obtained images of trademarks suspected of infringement from product images.

[0158] Step S1430: Output the mapping relationship data between the allegedly infringing trademark image and its corresponding legitimate trademark.

[0159] At this point, the mapping data between the allegedly infringing trademark images and their corresponding legitimate trademarks already exists in the allegedly infringing trademark image set A. infri The set is output as the result data, thus completing the detection of potentially infringing trademark images in the product images of the target product.

[0160] When the target product is a product that a merchant on an e-commerce platform initially intends to list, after detection by the technical solution of this application, if the number of allegedly infringing trademark images is greater than 0, that is, there is at least one allegedly infringing trademark image, the merchant's product listing process can be suspended, and the set A of the allegedly infringing trademark images can be removed. infri The image is sent to a manual review interface. Once the manual review interface confirms that the product image does not infringe on any rights, the product publishing process continues. If the number of allegedly infringing trademark images is 0, the merchant is allowed to directly publish the target product, thus enabling the target product to be published in the merchant's online store.

[0161] When the target product is obtained by the e-commerce platform's backend process from searching the online store's product database, if the number of allegedly infringing trademark images is greater than 0 (meaning at least one allegedly infringing trademark image exists), the corresponding target product can be temporarily removed from the platform, and an infringement notice message can be sent to the relevant merchant. The product can be restored after the merchant responds to the infringement notice message and successfully completes the appeal process. If the number of allegedly infringing trademark images is 0, no intervention is required, and the product infringement detection of other target products in the product database continues silently.

[0162] As can be seen from the above embodiments, after this application determines that the product image of the target product contains a trademark image suspected of infringement based on the characteristics of multiple modalities of the target product, it can output the corresponding results to play an alarm role, effectively maintain the healthy operation of the intellectual property protection mechanism of the e-commerce platform, thereby effectively maintaining the market order of the e-commerce platform, ensuring the legal operation of the e-commerce platform, and reducing platform risks.

[0163] Based on any of the above embodiments, the trademark recognition model of this application can be iteratively trained to a convergent state using a preset dataset. The dataset contains multiple training samples, each containing a single product image and text information of the product. Labels corresponding to legitimate trademarks in a preset trademark database are used as supervision labels for the model, indicating that the allegedly infringing trademark image in the product image is linked to the image. In each iteration, the product image and text information of a single training sample are used as input to the trademark recognition model. The model then obtains the confidence score and corresponding legitimate trademark label for each trademark image in the product image. The model loss is calculated using the supervision labels, and gradient updates are performed on the model if it has not converged, until the model is determined to have reached convergence. Thus, the trademark recognition model learns the ability to identify allegedly infringing trademark images from product images and text information of a target product, serving the needs of various embodiments of this application.

[0164] Please see Figure 13 To meet one of the purposes of this application, a product infringement detection device is provided, which is a functional embodiment of the product infringement detection method of this application. The device includes an image and text acquisition module 1100, a trademark extraction module 1200, a joint analysis module 1300, and an infringement determination module 1400, wherein: the image and text acquisition module 1100 is used to acquire product images and text information from the product information of the target product, the text information including the product description information of the target product; the trademark extraction module 1200 is used to extract a set of trademark images from the product images, the set of trademark images containing one or more trademark images; the joint analysis module 1300 is used to jointly determine the infringement probability of each trademark image based on the set of trademark images and the text information; the infringement determination module 1400 is used to determine the trademark images suspected of infringement in the set of trademark images based on the infringement probability.

[0165] Based on any of the above embodiments, the joint analysis module 1300 includes: a trademark feature extraction submodule, used to extract image features of each trademark image in the trademark image set; a text feature extraction submodule, used to extract text features of the text information; a multi-modal feature fusion submodule, used to fuse the text features with the image features to obtain fused features; and an infringement probability determination submodule, used to normalize the interaction features between the fused features and the text features to obtain the infringement probability corresponding to each trademark image.

[0166] Based on any of the above embodiments, the multi-modal feature fusion submodule includes: an initial fusion unit, used to apply an attention layer to perform initial feature fusion of the text features and image features to obtain initial features; an activation correction unit, used to perform multiple nonlinear activations on the image features to obtain corrected features; and a secondary fusion unit, used to perform secondary feature fusion of the initial features and the corrected features to obtain fused features.

[0167] Based on any of the above embodiments, the initial fusion unit includes: a vector input subunit, used to input the image features as a query vector and the text features as a key vector and a value vector into the attention layer; a weight extraction subunit, used by the attention layer to interact with and normalize the query vector and the key vector to obtain a weight matrix; and a feature generation subunit, used by the attention layer to match the value vector with the weight matrix to obtain initial features.

[0168] Based on any of the above embodiments, the activation correction unit includes: an initial correction subunit, used to perform a first nonlinear activation after fully connecting the image features to obtain an initial correction feature; and a secondary correction subunit, used to perform a second nonlinear activation after fully connecting the initial correction feature to obtain a final correction feature.

[0169] Based on any of the above embodiments, the infringement probability determination submodule includes: a shallow interaction unit, used to perform shallow feature interaction between the fused features and the text features to obtain shallow interaction features; a deep interaction unit, used to apply an attention layer to perform deep feature interaction on the shallow interaction features to obtain deep interaction features; a vector transformation unit, used to transform and pool the deep interaction features into a high-dimensional vector, such that each dimension corresponds to a trademark image; and a probability calculation unit, used to calculate and determine the infringement probability corresponding to each trademark image based on the high-dimensional vector.

[0170] Based on any of the above embodiments, the infringement determination module 1400 includes: a trademark mapping submodule, used to classify and map the image features of each trademark image in the trademark image set to determine the legal trademark corresponding to each trademark image; an image determination submodule, used to select the suspected infringing trademark image in the trademark image set according to the infringement probability corresponding to each trademark image; and a data output submodule, used to output the mapping relationship data between the suspected infringing trademark image and its corresponding legal trademark.

[0171] To address the aforementioned technical problems, embodiments of this application also provide computer equipment. For example... Figure 14 The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store a sequence of control information. When executed by the processor, the computer-readable instructions enable the processor to implement a product search category identification method. The processor provides computing and control capabilities to support the operation of the entire computer device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to execute the product infringement detection method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0172] In this embodiment, the processor is used to execute... Figure 13 The system contains the specific functions of each module and its sub-modules. The memory stores the program code and various data required to execute these modules or sub-modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / sub-modules in the product infringement detection device of this application. The server can call the server's program code and data to execute the functions of all sub-modules.

[0173] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the product infringement detection method of any embodiment of this application.

[0174] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0176] In summary, this application, in the process of trademark image infringement detection, combines the trademark image with the text information of the target product to more effectively determine whether a trademark image is suspected of infringement. This makes the identification of suspected infringing trademark images more accurate, avoids misjudgments, and ensures that the product images of the target product can be accurately determined to contain unauthorized trademarks, thereby maintaining the market order of e-commerce platforms and effectively safeguarding the operation of the intellectual property protection mechanism of e-commerce platforms.

[0177] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0178] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for detecting product infringement, characterized in that, Includes the following steps: Obtain product images and text information from the product information of the target product, wherein the text information includes the product description information of the target product; A set of trademark images is extracted from the product images. The set of trademark images contains multiple trademark images, which are obtained by recognizing and extracting the product images using an object detection model. Determining the infringement probability of each trademark image based on the trademark image set and the text information includes: extracting image features of each trademark image in the trademark image set; extracting text features of the text information; fusing the text features and the image features to obtain fused features; and normalizing based on the interaction features between the fused features and the text features to obtain the infringement probability corresponding to each trademark image. Determining the suspected infringing trademark images in the trademark image set based on the infringement probability includes: classifying and mapping the image features of each trademark image in the trademark image set to determine the corresponding legitimate trademark; selecting the suspected infringing trademark images in the trademark image set based on the infringement probability corresponding to each trademark image; and outputting the mapping relationship data between the suspected infringing trademark images and their corresponding legitimate trademarks. The text features and image features are fused to obtain fused features, including: applying an attention layer to perform initial feature fusion of the text features and image features to obtain initial features; applying multiple nonlinear activations to the image features through a squeezing gate to obtain modified features; and performing secondary feature fusion of the initial features and the modified features to obtain fused features. The infringement probability corresponding to each trademark image is obtained by normalizing the interaction features between the fused features and the text features, including the following steps: performing shallow feature interaction between the fused features and the text features to obtain shallow interaction features; applying an attention layer to perform deep feature interaction on the shallow interaction features to obtain deep interaction features; converting and pooling the deep interaction features into a high-dimensional vector, with each dimension corresponding to a trademark image; and calculating and determining the infringement probability corresponding to each trademark image based on the high-dimensional vector.

2. The method for detecting product infringement according to claim 1, characterized in that, The text features and image features are initially fused using an attention layer to obtain initial features, including the following steps: The image features are used as query vectors, and the text features are used as key and value vectors, which are then input into the attention layer. The attention layer interacts with and normalizes the query vector and the key vector to obtain a weight matrix; The attention layer matches the value vector with the weight matrix to obtain initial features.

3. The method for detecting product infringement according to claim 1, characterized in that, The image features are subjected to multiple nonlinear activations to obtain corrected features, including the following steps: The image features are fully connected and then subjected to a first nonlinear activation to obtain the initial corrected features. The initial modified features are fully connected and then subjected to a second nonlinear activation to obtain the final modified features.

4. A product infringement detection device, characterized in that, include: The image and text acquisition module is used to acquire product images and text information from the product information of the target product, wherein the text information includes the product description information of the target product; The trademark extraction module is used to extract a set of trademark images from the product images. The set of trademark images contains multiple trademark images, which are obtained by recognizing and extracting the product images using an object detection model. The joint analysis module is used to jointly determine the infringement probability of each trademark image based on the trademark image set and the text information. It includes: a trademark feature extraction submodule for extracting image features of each trademark image in the trademark image set; a text feature extraction submodule for extracting text features from the text information; a multi-modal feature fusion submodule for fusing the text features and image features to obtain fused features; and an infringement probability determination submodule for normalizing the interaction features between the fused features and the text features to obtain the infringement probability corresponding to each trademark image. The infringement determination module is used to determine the trademark images suspected of infringement in the trademark image set based on the infringement probability, including: classifying and mapping the image features of each trademark image in the trademark image set to determine the corresponding legitimate trademarks; selecting the trademark images suspected of infringement in the trademark image set based on the infringement probability corresponding to each trademark image; and outputting the mapping relationship data between the suspected infringing trademark images and their corresponding legitimate trademarks. The multimodal feature fusion submodule includes: an initial fusion unit, used to apply an attention layer to perform initial feature fusion of the text features and image features to obtain initial features; an activation correction unit, used to perform multiple nonlinear activations of the image features through a squeezing gate to obtain corrected features; and a secondary fusion unit, used to perform secondary feature fusion of the initial features and the corrected features to obtain fused features. The infringement probability determination submodule includes: a shallow interaction unit, used to perform shallow feature interaction between the fused features and the text features to obtain shallow interaction features; a deep interaction unit, used to apply an attention layer to perform deep feature interaction on the shallow interaction features to obtain deep interaction features; a vector transformation unit, used to transform and pool the deep interaction features into a high-dimensional vector, such that each dimension corresponds to a trademark image; and a probability calculation unit, used to calculate and determine the infringement probability corresponding to each trademark image based on the high-dimensional vector.

5. The product infringement detection device according to claim 4, characterized in that, The initial fusion unit includes: A vector input subunit is used to input the image features as a query vector and the text features as key and value vectors into the attention layer. The weight extraction subunit is used by the attention layer to interact with and normalize the query vector and the key vector to obtain a weight matrix; A feature generation subunit is used to obtain initial features by matching the value vector with the weight matrix from the attention layer.

6. The product infringement detection device according to claim 4 or 5, characterized in that, The activation correction unit includes: An initial correction subunit is used to perform a first nonlinear activation on the fully connected image features to obtain the initial corrected features; The secondary correction subunit is used to fully connect the initial correction feature and then perform a second nonlinear activation to obtain the final correction feature.

7. A computer device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 3, which, when invoked by a computer, executes the steps included in the corresponding method.

Images