Image-text model training method, commodity identification method, equipment and medium

Through the graphic and text model training method, the naming entity information and picture information of the product are extracted and the similarity is calculated, which solves the problem of insufficient product recognition efficiency and accuracy in the prior art, and achieves more efficient product matching and behavioral data statistics.

CN119068278APending Publication Date: 2024-12-03XIAOHONGSHU TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310640851.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-01
Publication Date
2024-12-03

Smart Images

  • Figure CN119068278A_ABST
    Figure CN119068278A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of artificial intelligence, in particular to an image-text model training method, a commodity recognition method, equipment and a medium. The image-text model training method comprises the steps of obtaining training data, wherein the training data comprises texts and pictures related to a plurality of commodities; for each commodity, extracting named entity information from the text, and extracting picture information from the picture; the named entity information and the picture information of the multiple commodities are utilized to train a pre-constructed model, a trained image-text model is obtained, the pre-constructed model comprises a first model, a second model and a third model, the named entity information of the two commodities is input into the first model to calculate text similarity, and the text similarity is obtained; and inputting the picture information of the two commodities into a second model to calculate a picture similarity, and inputting the named entity information and the picture information of the two commodities into a third model to calculate a fusion similarity. Through multi-mode information fusion, the commodity identification efficiency and accuracy can be further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a graphic model training method, a commodity recognition method, equipment and medium. Background Art

[0002] Users will publish and share user-generated content (UGC) with other users on the Internet, and the content may be about a certain product.

[0003] In order to count the exposure, clicks, likes, collections and other behavioral data of products in these user-generated contents, it is necessary to first identify the products that appear in the user-generated contents. Summary of the invention

[0004] In order to solve the above problems, the present application provides a graphic model training method, a commodity recognition method, a device and a medium.

[0005] The first aspect of the present application discloses a method for training a graphic model for an electronic device, the method comprising:

[0006] Acquire training data, wherein the training data includes text and images related to a plurality of commodities;

[0007] For each product, extract the named entity information of the product from the text related to the product, and extract the image information of the product from the image related to the product;

[0008] Using the named entity information and the image information of the multiple commodities to train a pre-built model to obtain the trained image-text model,

[0009] The pre-built model includes a first model, a second model and a third model, the named entity information of two of the multiple commodities is input into the first model to calculate the text similarity of the two commodities, the image information of two of the multiple commodities is input into the second model to calculate the image similarity of the two commodities, and the named entity information and the image information of two of the multiple commodities are input into the third model to calculate the fusion similarity of the two commodities, and the third model fuses the named entity information and the image information of the two commodities.

[0010] Optionally, the named entity information includes one or more of a brand, a series, and a category of a product, and the image information includes a cropped image corresponding to the product detection frame.

[0011] Optionally, the first model is a text Transformer model, the second model is an image Transformer model, and the third model is a fusion Transformer model.

[0012] A second aspect of the present application discloses a commodity identification method for an electronic device, the method comprising:

[0013] Receiving data to be processed and commodity library data, wherein the data to be processed includes text and pictures related to the commodity to be processed, and the commodity library data includes text and pictures related to commodities in multiple commodity libraries;

[0014] The data to be processed and the commodity library data are used as inputs of the graphic model, and a matching result between the commodity to be processed and at least a part of the commodity library commodities in the plurality of commodity library commodities is outputted.

[0015] Wherein, the graphic model is trained according to the method described in the first aspect of the present application.

[0016] Optionally, the method further comprises:

[0017] For the commodity to be processed, extracting the named entity information of the commodity to be processed from the text related to the commodity to be processed, and extracting the image information of the commodity to be processed from the image related to the commodity to be processed;

[0018] Based on the named entity information and the image information of the to-be-processed product, at least a portion of the product library products are recalled from the multiple product library products as candidate product library products, wherein each candidate product library product has the named entity information and the image information of the candidate product library product.

[0019] Optionally, the method further comprises:

[0020] Filtering the candidate product library products recalled based on the named entity information of the product to be processed, wherein the candidate product library products that do not match the brand, series and category of the product to be processed are removed;

[0021] Inputting the named entity information of the to-be-processed product and each of the filtered candidate product library products into the first model to calculate the text similarity of the two products;

[0022] Based on the text similarity, a matching product library product that matches the product text to be processed is determined, and an unmatched product library product that does not match the product text to be processed is excluded.

[0023] Optionally, the method further comprises:

[0024] Filtering the candidate product library products recalled based on the image information of the product to be processed, wherein the named entity information of the product and the candidate product library products are identified from the image information of the product and the candidate product library products respectively, and removing the candidate product library products that do not match the brand, series and category of the product to be processed;

[0025] Inputting the image information of the product to be processed and each of the filtered candidate product library products into the second model to calculate the image similarity of the two products;

[0026] Based on the image similarity, a matching product library product that matches the image of the product to be processed is determined, and an unmatched product library product that does not match the image of the product to be processed is excluded.

[0027] Optionally, the method further comprises:

[0028] For the to-be-processed product for which no matching product in the product library is determined, inputting the named entity information and the image information of the to-be-processed product and each of the excluded candidate product library products into the third model to calculate the fusion similarity of the two products;

[0029] Based on the fusion similarity, a matching product library product that is fused and matched with the product to be processed is determined, and a non-matching product library product that is not fused and matched with the product to be processed is excluded.

[0030] Optionally, in a case where there are multiple matching product library products that match the text of the product to be processed determined based on the text similarity, the image information of the product to be processed and each matching product library product is input into the second model to calculate the image similarity of the two products, and based on the text similarity and the image similarity, a matching product library product that matches the text of the product to be processed is determined.

[0031] Optionally, in the case that there are multiple matching product library products that match or merge match the image of the product to be processed based on the image similarity or the fusion similarity, the image similarity or the fusion similarity of each matching product library product is updated based on whether the brand and category of each matching product library product matches the brand and category of the product to be processed, and based on the updated image similarity or the fusion similarity, a matching product library product that matches or merge match the image of the product to be processed is determined.

[0032] The third aspect of the present application discloses a graphic model training device, the device comprising:

[0033] An acquisition module is configured to acquire training data, wherein the training data includes text and pictures related to a plurality of commodities;

[0034] An extraction module is configured to extract, for each product, named entity information of the product from the text related to the product, and image information of the product from the image related to the product;

[0035] A training module is configured to train a pre-built model using the named entity information and the image information of the plurality of commodities to obtain the trained image-text model,

[0036] The pre-built model includes a first model, a second model and a third model, the named entity information of two of the multiple commodities is input into the first model to calculate the text similarity of the two commodities, the image information of two of the multiple commodities is input into the second model to calculate the image similarity of the two commodities, and the named entity information and the image information of two of the multiple commodities are input into the third model to calculate the fusion similarity of the two commodities, and the third model fuses the named entity information and the image information of the two commodities.

[0037] A fourth aspect of the present application discloses a commodity identification device, the device comprising:

[0038] A receiving module, configured to receive data to be processed and commodity library data, wherein the data to be processed includes text and pictures related to the commodity to be processed, and the commodity library data includes text and pictures related to commodities in multiple commodity libraries;

[0039] an identification module configured to use the data to be processed and the commodity library data as inputs of the graphic model, and output a matching result between the commodity to be processed and at least a part of the commodity library commodities in the plurality of commodity library commodities,

[0040] Wherein, the graphic model is trained according to the method described in the first aspect of the present application.

[0041] The fifth aspect of the present application discloses an electronic device, which includes a memory storing computer-executable instructions and a processor; when the computer-executable instructions are executed by the processor, the electronic device implements the method described in the first aspect of the present application, or implements the method described in the second aspect of the present application.

[0042] The sixth aspect of the present application discloses a computer-readable medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect of the present application, or to implement the method described in the second aspect of the present application.

[0043] The seventh aspect of the present application discloses a computer program product, including a computer program, which, when executed by a processor, implements the method according to the first aspect of the present application, or implements the method according to the second aspect of the present application.

[0044] Compared with the existing technology, this application can further improve the efficiency and accuracy of product identification through multi-modal information fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present application;

[0046] Figure 2 is a flowchart of a method for training a graphic model according to an embodiment of the present application;

[0047] Figure 3 is a schematic diagram of a method for training a graphic model according to an embodiment of the present application;

[0048] Figure 4 is a flow chart of a commodity identification method according to an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] The present application will be further described below in conjunction with specific embodiments and accompanying drawings. It will be appreciated that the specific embodiments described herein are only intended to explain the present application and are not intended to limit the present application. In addition, for ease of description, only some but not all structures or processes related to the present application are shown in the accompanying drawings. It should be noted that in this specification, similar reference numerals and letters represent similar items in the following accompanying drawings.

[0051] It should be understood that although the terms "first", "second", etc. may be used herein to describe various features, these features should not be limited by these terms. These terms are used only to distinguish and should not be understood as indicating or implying relative importance. For example, a first feature may be referred to as a second feature, and similarly a second feature may be referred to as a first feature without departing from the scope of the exemplary embodiments.

[0052] In order to make the objectives, technical solutions and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0053] Figure 1 : is a schematic diagram of an application scenario according to an embodiment of the present application. In this application scenario, the device involved may include a model training device for training a graphic model, and the model training device may be a server or a terminal. Figure 1 Take the model training device as the first server 101 as an example. On the first server 101, the pre-built model can be trained based on the training data containing text and pictures to obtain a trained picture and text model.

[0054] like Figure 1 As shown, the training data may be collected in advance and stored in a database.

[0055] like Figure 1 As shown, the device involved in the application scenario may also include a processing device for processing the data to be processed through the graphic model, and the processing device may be a server or a terminal. Figure 1 For example, the processing device is the second server 102. After the graph model is trained based on the training data on the first server 101, the graph model on the first server 101 is deployed on the second server 102, and the data to be processed is processed on the second server 102 by the graph model.

[0056] like Figure 1 As shown, the device involved in the application scenario may also include a terminal 103 , and the terminal 103 may send the data to be processed to the second server 102 . The second server 102 processes the data to be processed through a graphic model and returns the result to the terminal 103 .

[0057] Among them, the client 103 can be one or more, and the number of the client 103 should not be used as a limiting condition of the present application. The client 103 can be a computer device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., which has computing power, input and output functions, and network connection functions. The servers 101 and 102 can be a single server device, or a server cluster, a cloud server, etc., which can provide application services to the outside.

[0058] The technical solution of the present application and how the technical solution of the present application solves the technical problem are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0059] Figure 2It is a flowchart of a method for training a graphic model according to an embodiment of the present application.

[0060] like Figure 2 As shown, in S201, training data is obtained, and the training data includes texts and pictures related to multiple commodities.

[0061] As an implementation method, the training data may be derived from a large amount of user-generated content, wherein each user-generated content includes text content describing one or more products (such as the title and body of the user-generated content, text lines generated using optical character recognition (OCR) technology or automatic speech recognition (ASR) technology), and includes image content containing background information and / or products.

[0062] As an implementation method, the training data may be derived from product library data, wherein the product library may include a large number of product library products, which may also be referred to as Standard Product Units (SPUs). The product library may be pre-established and maintained, the product library data may be derived from official product content, and each product library product may have a product name and a main product image of the product library product.

[0063] As an implementation, a portion of the training data may be derived from a large amount of user-generated content, and another portion of the training data may be derived from product library data.

[0064] It is worth noting that the acquisition, storage, use, and processing of data in the technical solutions disclosed in the embodiments of this application are in compliance with the relevant provisions of national laws and regulations.

[0065] In S202 , for each product, named entity information of the product is extracted from text related to the product, and image information of the product is extracted from images related to the product.

[0066] As an implementation, based on the brand, series and category dictionary and / or Name Entity Recognition (NER) technology, the named entity information of the product is extracted from the text related to the product, and the named entity information may include one or more of the brand, series and category of the product. As an example, the text related to the product may be "A brand B series day cream", and the named entity information of the product may be "A brand", "B series" and "day cream".

[0067] As an implementation method, the image information may include a cropped image corresponding to the product detection frame. The number of the cropped images may be one or more. Those skilled in the art will appreciate that the extraction of the above image information may be achieved using the product recognition technology based on image detection in the prior art, which will not be described in detail here.

[0068] As an implementation mode, the named entity information of the product name of the product in the product library can be pre-extracted, and the main image of the product in the product library can be directly used as the image information of the product in the product library.

[0069] In S203, the pre-built model is trained using the named entity information and image information of the multiple products to obtain a trained image-text model.

[0070] The pre-built model includes a first model, a second model and a third model, the named entity information of two commodities among the multiple commodities is input into the first model to calculate the text similarity of the two commodities, the image information of two commodities among the multiple commodities is input into the second model to calculate the image similarity of the two commodities, and the named entity information and image information of two commodities among the multiple commodities are input into the third model to calculate the fusion similarity of the two commodities, and the third model fuses the named entity information and image information of the two commodities.

[0071] As an implementation, the first model is a text Transformer model, the second model is an image Transformer model, and the third model is a fusion Transformer model. As an example, the first model is a BERT model, the second model is a visual BERT model, and the third model is a multimodal BERT model.

[0072] As an implementation method, the training data can be manually annotated, that is, it is manually determined that two products match (that is, the text similarity, image similarity and / or fusion similarity of the two products are high), and two products do not match (that is, the text similarity, image similarity and / or fusion similarity of the two products are low). The matching product pairs constitute positive examples, while the non-matching product pairs constitute negative examples, and the named entity information and image information of the positive or negative examples are input into a pre-built model to calculate the text similarity, image similarity and fusion similarity of the two products in the product pair.

[0073] As an implementation method, the first model is trained using pairs of products with high or low text similarity, the second model is trained using pairs of products with high or low image similarity, and the third model is trained using pairs of products with average text similarity and image similarity but high or low fusion similarity. At this time, the output results of the first to third models can be compared with the manual annotation results, and the parameters of the first to third models can be updated according to the comparison results.

[0074] Based on this, considering that manual identification of products will first compare the text content of two products separately or the image content of two products separately, and then comprehensively compare the text content and image content of the two products, the pre-built model can learn the alignment relationship between the original text and image of the product and the official text and image, and then learn the information relied on when manually identifying products, thereby achieving the effect of product identification.

[0075] like Figure 2 As shown, according to an embodiment of the present application, the graphic model training method is performed in the order of the above steps, but those skilled in the art will understand that this should not be regarded as a limitation of the present application. For example, there may be two or more steps that can be performed simultaneously or sequentially and the order of the two steps can be interchanged.

[0076] The following combination Figure 3 right Figure 2 The graphic training method shown is explained. Figure 3 It is a schematic diagram of a method for training a graphic model according to an embodiment of the present application.

[0077] like Figure 3 As shown, the input 302 received by the first model 301 includes the named entity information of the product from the user-generated content (ie, Name 1) and the named entity information of the product from the product library (ie, Spuname 1). Using the contrastive learning method, the first model 301 learns the text similarity of the product pair.

[0078] The input 304 received by the second model 303 includes the image information of the product from the user-generated content (ie, Crop1) and the image information of the product from the product library (ie, Spucrop 1). The second model 303 learns the image similarity of the product pair by contrastive learning.

[0079] For product pairs that are difficult to judge by the first model 301 and the second model 303, the named entity information (namely, Names and Spunames) and image information (namely, Crops and Spucrops) of these product pairs are used as input 306 of the third model 305. The third model 305 learns the fusion similarity of each product pair by contrastive learning.

[0080] The first model 301 has a text loss function, the second model 302 has an image loss function, and the third model 303 has a fusion loss function, wherein these loss functions may be, for example, cross entropy loss functions. Next, through iterative training, when the loss function values ​​of the first to third models 301, 303, 305 are all less than or equal to their respective loss thresholds, the model training is stopped, or when the number of iterations reaches a number threshold, the model training is stopped.

[0081] According to an embodiment of the present application, a commodity identification method is also included. Figure 4 It is a flowchart of a commodity identification method according to an embodiment of the present application.

[0082] like Figure 4 As shown, in S401, data to be processed and commodity library data are received, the data to be processed includes text and pictures related to the commodity to be processed, and the commodity library data includes text and pictures related to commodities in multiple commodity libraries.

[0083] As an implementation method, the data to be processed may come from a large amount of user-generated content, wherein each user-generated content includes text content describing one or more products (such as the title and body of the user-generated content, text lines generated using OCR technology or ASR technology), and includes image content containing background information and / or products. It is necessary to identify which product library product in the product library the product appearing in the user-generated content belongs to, so that the behavior data of the product library product can be subsequently counted.

[0084] In S402, the data to be processed and the commodity library data are used as inputs of the graphic model, and a matching result between the commodity to be processed and at least a part of the commodity library commodities in the plurality of commodity library commodities is output.

[0085] As an implementation mode, for the commodity to be processed, the named entity information of the commodity to be processed is extracted from the text related to the commodity to be processed, and the image information of the commodity to be processed is extracted from the image related to the commodity to be processed. Based on the named entity information and the image information of the commodity to be processed, at least a part of the commodity library commodities are recalled from multiple commodity library commodities as candidate commodity library commodities, wherein each candidate commodity library commodity has the named entity information and the image information of the candidate commodity library commodity.

[0086] As described above, based on the brand, series and category dictionary and / or NER technology, the named entity information of the product to be processed is extracted from the text related to the product to be processed, and the named entity information may include one or more of the brand, series and category of the product to be processed. The image information may include a cropped image corresponding to the detection frame of the product to be processed, and the number of the cropped images may be one or more. The product name of the product in the candidate product library may be pre-extracted with the named entity information, and the main image of the product in the candidate product library may be directly used as the image information of the product in the candidate product library.

[0087] Recalling candidate product library products based on the named entity information of the product to be processed may be based on one of the brand, series and category of the product to be processed. As an example, in the case where the named entity information of the product to be processed is "Brand A", "Series B" and "Day Cream", the candidate product library products are recalled based on "Brand A", "Series B" and "Day Cream", and the recalled candidate product library products may be "Brand A E Series Facial Mask", "Brand F B Series Essence", "Brand G H Series Day Cream", etc.

[0088] Recalling products from the candidate product library based on the image information of the product to be processed can be achieved by using image retrieval technology in the prior art, which will not be described in detail here.

[0089] As an implementation method, the candidate product library products recalled based on the named entity information of the product to be processed are filtered, wherein the candidate product library products that do not match the brand, series and category of the product to be processed are removed. The named entity information of each candidate product library product in the product to be processed and the filtered candidate product library products is input into the first model to calculate the text similarity of the two products. Based on the text similarity, the matching product library products that match the text of the product to be processed are determined, and the unmatched product library products that do not match the text of the product to be processed are excluded.

[0090] Since the candidate product library products are recalled based on one of the brand, series and category of the product to be processed, although the scope of the recall can be expanded, there is a lot of noise in the recall process, and there are many obviously mismatched candidate product library products in the recalled candidate product library products.

[0091] Continuing with the above example, when the named entity information of the product to be processed is "Brand A", "Series B" and "Day Cream", the candidate product library products are recalled based on "Brand A", "Series B" and "Day Cream", respectively. The recalled candidate product library products can be "Brand A E Series Mask", "Brand F B Series Essence", "Brand G H Series Day Cream", etc. Among them, the candidate product library product "Brand F B Series Essence" and the candidate product library product "Brand G H Series Day Cream" are obviously inconsistent with the brand "Brand A" of the product to be processed, and can be removed. Although the candidate product library product "Brand A E Series Mask" is consistent with the brand "Brand A" of the product to be processed, it is obviously inconsistent with the series "B Series" and category "Day Cream" of the product to be processed, and can also be removed.

[0092] Furthermore, when determining whether the brands of the candidate product library products match the products to be processed, the brand synonym table is combined. For example, although the Chinese brand and the English brand of the same brand are not consistent, the two are substantially matched.

[0093] Furthermore, when determining whether the product in the candidate product library matches the series or category of the product to be processed, the series or category exclusive table is combined. For example, the category "facial mask" and the category "day cream" are mutually exclusive, so the two do not match, while the category "night cream" and the category "day cream" are not mutually exclusive, so the two match.

[0094] Furthermore, for the candidate product library products recalled based on the named entity information of the product to be processed, only the candidate product library products that do not match the brand and category of the product to be processed are removed. Considering that the official description of the series is often complicated, and users often omit or change the description of the series when describing the product, it is difficult to obtain an ideal matching effect when determining whether the candidate product library products match the series of the product to be processed. Therefore, it is also possible not to match the candidate product library products with the series of the product to be processed.

[0095] After unmatched candidate product library products are removed from the recalled candidate product library products, the named entity information of the product to be processed and each candidate product library product in the filtered candidate product library products is input into the first model to calculate the text similarity of the two products.

[0096] The named entity information of the commodity to be processed can constitute the commodity name of the commodity to be processed, wherein the commodity name can be in the form of a triple (brand, series, category) or in the form starting from the brand and ending with the category, and each candidate commodity library commodity also has the commodity name of the candidate commodity library commodity, and the two commodity names can constitute a text pair. The first model performs feature extraction and calculates the similarity of the text pair (ie, text similarity), thereby calculating the text similarity of the two commodities.

[0097] Next, based on the text similarity, the matching product library products that match the product text to be processed are determined, and the unmatched product library products that do not match the product text to be processed are excluded.

[0098] For example, a candidate product library item with the highest text similarity is selected as the matching product library item, or multiple candidate product library items with text similarities exceeding a text similarity threshold are selected as the matching product library items.

[0099] Further, for multiple candidate product library products whose text similarity exceeds the text similarity threshold, when the text similarity difference between the candidate product library product ranked lower in text similarity and the candidate product library product ranked first in text similarity exceeds the difference threshold, the candidate product library product ranked first in text similarity is selected as the matching product library product. As an example, assuming that the text similarity threshold is 0.75 and the difference threshold is 0.1, for example, there is a first-ranked candidate product library product with a text similarity of 0.99 and a second-ranked candidate product library product with a text similarity of 0.8. Although the text similarities of the two candidate product library products both exceed the text similarity threshold, the difference in text similarity between the second-ranked candidate product library product and the first-ranked candidate product library product exceeds the difference threshold. At this time, it can be considered that the first-ranked candidate product library product has a better match, while the second-ranked candidate product library product has a poorer match, so the first-ranked candidate product library product can be directly selected as the matching product library product.

[0100] Similarly, a candidate product library product with the lowest text similarity ranking is selected as the unmatched product library product, or multiple candidate product library products whose text similarity does not exceed another text similarity threshold are selected as the unmatched product library products. Further, for multiple candidate product library products whose text similarity does not exceed another text similarity threshold, when the text similarity difference between the candidate product library product with the highest text similarity ranking and the candidate product library product with the lowest text similarity ranking exceeds the difference threshold, the candidate product library product with the lowest text similarity ranking is selected as the unmatched product library product.

[0101] In this way, the product pairs that are obviously matched and obviously mismatched can be selected, while the product pairs that are suspected to be matched are left. For example, due to the missing of some information in the product name triple (brand, series, category) of the product to be processed, it can be similar to the product texts of multiple candidate product libraries.

[0102] As an implementation method, the candidate product library products recalled based on the image information of the product to be processed are filtered, wherein the named entity information of the product and the candidate product library products are identified from the image information of the product and the candidate product library products, respectively, and the candidate product library products that do not match the brand, series and category of the product to be processed are removed. The image information of each candidate product library product in the product to be processed and the filtered candidate product library products is input into the second model to calculate the image similarity of the two products. Based on the image similarity, the matching product library products that match the image of the product to be processed are determined, and the unmatched product library products that do not match the image of the product to be processed are excluded.

[0103] Since the image retrieval technology in the existing technology is used to recall the products in the candidate product library based on the picture information of the products to be processed, considering that the image retrieval technology usually adopts a dual-tower based model, it often has defects in distinguishing fine-grained image differences. Therefore, there is also a lot of noise in the recall process, and there are also many obviously mismatched candidate product library products in the recalled candidate product library products.

[0104] Those skilled in the art will appreciate that the identification of the above-mentioned named entity information can be achieved by utilizing OCR technology in combination with brand, series and category dictionaries and / or NER technology, which will not be elaborated herein.

[0105] As an example, recalling the products in the candidate product library based on the cutouts of the products to be processed, the named entity information of the products to be processed identified from the cutouts of the products to be processed may be "Brand A", "Series B" and "Day Cream", and the named entity information of the products in the candidate product library identified from the main images of the recalled products in the candidate product library may be "Brand A E Series Mask", "Brand F B Series Essence", and "Brand G H Series Day Cream", etc. Among them, the candidate product library products "Brand F B Series Essence" and the candidate product library product "Brand G H Series Day Cream" are obviously inconsistent with the brand "Brand A" of the products to be processed and can be removed, while the candidate product library product "Brand A E Series Mask" is consistent with the brand "Brand A" of the products to be processed, but is obviously inconsistent with the series "B Series" and category "Day Cream" of the products to be processed and can also be removed.

[0106] Furthermore, when determining whether the brands of the candidate product library products match the products to be processed, the brand synonym table is combined. For example, although the Chinese brand and the English brand of the same brand are not consistent, the two are substantially matched.

[0107] Furthermore, when determining whether the product in the candidate product library matches the series or category of the product to be processed, the series or category exclusive table is combined. For example, the category "facial mask" and the category "day cream" are mutually exclusive, so the two do not match, while the category "night cream" and the category "day cream" are not mutually exclusive, so the two match.

[0108] Considering that the cropped images of the products to be processed and the main images of the products in the candidate product library often retain the product series completely, a more ideal matching effect can be obtained when determining whether the series of the products in the candidate product library match the products to be processed. Therefore, compared with text matching, image matching can not only match the brands and categories of the products in the candidate product library and the products to be processed, but also match the series of the products in the candidate product library and the products to be processed.

[0109] After unmatched candidate product library products are removed from the recalled candidate product library products, the image information of the product to be processed and each candidate product library product in the filtered candidate product library products is input into the second model to calculate the image similarity of the two products.

[0110] The cropped image of the product to be processed and the main image of each candidate product library product can form an image pair. The second model performs feature extraction and calculates the similarity of the image pair (ie, image similarity), thereby calculating the image similarity of the two products.

[0111] Next, based on the image similarity, the matching product library products that match the image of the product to be processed are determined, and the unmatched product library products that do not match the image of the product to be processed are excluded.

[0112] For example, a candidate product library product with the highest image similarity is selected as the matching product library product, or multiple candidate product library products with image similarities exceeding an image similarity threshold are selected as the matching product library products.

[0113] Further, for multiple candidate product library products whose image similarity exceeds the image similarity threshold, when the image similarity difference between the candidate product library product ranked lower in image similarity and the candidate product library product ranked first in image similarity exceeds the difference threshold, the candidate product library product ranked first in image similarity is selected as the matching product library product. As an example, assuming that the image similarity threshold is 0.75 and the difference threshold is 0.1, for example, there is a first-ranked candidate product library product with an image similarity of 0.99 and a second-ranked candidate product library product with an image similarity of 0.8. Although the image similarity of the two candidate product library products exceeds the image similarity threshold, the difference in image similarity between the second-ranked candidate product library product and the first-ranked candidate product library product exceeds the difference threshold. At this time, it can be considered that the first-ranked candidate product library product has a better matching degree, while the second-ranked candidate product library product has a poorer matching degree, so the first-ranked candidate product library product can be directly selected as the matching product library product.

[0114] Similarly, a candidate product library product with the lowest image similarity ranking is selected as the unmatched product library product, or multiple candidate product library products whose image similarity does not exceed another image similarity threshold are selected as the unmatched product library products. Further, for multiple candidate product library products whose image similarity does not exceed another image similarity threshold, when the image similarity difference between the candidate product library product with the highest image similarity ranking and the candidate product library product with the lowest image similarity ranking exceeds the difference threshold, the candidate product library product with the lowest image similarity ranking is selected as the unmatched product library product.

[0115] In this way, the product pairs that are obviously matched and obviously mismatched can be selected, while the product pairs that are suspected to be matched are left. For example, since the cropped image of the product to be processed is a partial image, it can be similar to the product images of multiple candidate product libraries.

[0116] As an implementation mode, for the pending product for which the matching product library product is not determined, the named entity information and image information of the pending product and each candidate product library product in the excluded candidate product library products are input into the third model to calculate the fusion similarity of the two products. Based on the fusion similarity, the matching product library product that is fused and matched with the pending product is determined, and the unmatched product library products that are not fused and matched with the pending product are excluded.

[0117] For example, due to the missing of some information in the product name triplet (brand, series, category) of a certain product to be processed, the first model cannot determine the matching product in the product library (but the non-matching product library products may be excluded), and the cropped image of the product to be processed cannot determine the matching product in the product library in the second model (but the non-matching product library products may be excluded), then the named entity information and image information of the product to be processed and each candidate product library product in the excluded candidate product library products can be input into the third model to calculate the fusion similarity of the two products.

[0118] As an example, the product name of the product to be processed may be "Brand A, Series B", and two suspected matching candidate product library products are obtained through the first model. The product name of one candidate product library product may be "Brand A, Series B Essence Cream", and the product name of the other candidate product library product may be "Brand A, Series B Essence Liquid". The cropped image of the product to be processed may be a partial image of a small brown bottle, and two suspected matching candidate product library products are also obtained through the second model. These four suspected matching candidate product library products may or may not be repeated, but after deduplication, the product to be processed and each suspected matching candidate product library product form a product pair. The named entity information and image information of each product pair can be input into the third model for feature extraction and fusion and the fusion similarity of each product pair is calculated to determine whether there is a matching product pair.

[0119] Next, based on the fusion similarity, the matching product library products that are fused and matched with the product to be processed are determined, and the unmatched product library products that are not fused and matched with the product to be processed are excluded.

[0120] For example, a candidate product library item with the highest fusion similarity is selected as the matching product library item, or multiple candidate product library items with fusion similarities exceeding a fusion similarity threshold are selected as the matching product library items.

[0121] Further, for multiple commodity library products whose fusion similarity exceeds the fusion similarity threshold, when the fusion similarity difference between the candidate commodity library product ranked lower in fusion similarity and the candidate commodity library product ranked first in fusion similarity exceeds the difference threshold, the candidate commodity library product ranked first in fusion similarity is selected as the matching commodity library product. As an example, assuming that the fusion similarity threshold is 0.75 and the difference threshold is 0.1, for example, there is a candidate commodity library product ranked first in fusion similarity of 0.99, and the fusion similarity of the candidate commodity library product ranked second in fusion similarity is 0.8, although the fusion similarity of the two candidate commodity library products exceeds the fusion similarity threshold, the difference in fusion similarity between the candidate commodity library product ranked second in fusion similarity and the candidate commodity library product ranked first in fusion similarity exceeds the difference threshold, at this time, it can be considered that the matching degree of the candidate commodity library product ranked first is better, while the matching degree of the candidate commodity library product ranked second in fusion similarity is poor, so the candidate commodity library product ranked first in fusion similarity can be directly selected as the matching commodity library product.

[0122] For the pending product that has not been determined to match the product library, it is determined that there is no product library product matching the pending product in the product library. The pending product may be a newly added product or an unrecognizable product.

[0123] As an implementation mode, when there are multiple matching product library products that match the text of the product to be processed determined based on text similarity, the image information of the product to be processed and each matching product library product is input into the second model to calculate the image similarity of the two products, and based on the text similarity and the image similarity, a matching product library product that matches the text of the product to be processed is determined.

[0124] Since the text similarity threshold is used for screening, there may be multiple matching product library products that meet the threshold requirements. As an example, assume that the text similarity threshold is 0.75, the text similarity of the first matching product library product is 0.9, and the text similarity of the second matching product library product is 0.85, both of which meet the threshold requirements. However, the image similarity of the first matching product library product is 0.7, and the image similarity of the second matching product library product is 0.9. Then the comprehensive similarity of the first matching product library product (such as a simple sum of text similarity and image similarity) is 1.6, and the comprehensive similarity of the second matching product library product is 1.75. At this time, the most matching product library product should be the second matching product library product.

[0125] As an implementation mode, when there are multiple matching product library products that match the image of the product to be processed based on the image similarity, the image similarity of each matching product library product is updated based on whether the brand and category of each matching product library product match the brand and category of the product to be processed, and based on the updated image similarity, a matching product library product that matches the image of the product to be processed is determined.

[0126] Since the image similarity threshold is used for screening, there may be multiple matching product library products that meet the threshold requirements. As an example, assuming that the image similarity threshold is 0.75, the image similarity of the first matching product library product is 0.8, and the image similarity of the second matching product library product is 0.85, both of which meet the threshold requirements. However, the product name of the first matching product library product can be "A Brand B Series Essence", the product name of the second matching product library product can be "A Brand B Series Mask", and the product name of the product to be processed can be "A Brand Essence", then the brand and category of the first matching product library product match the brand and category of the product to be processed, that is, the text side of the product to be processed has a series missing phenomenon, the image similarity of the first matching product library product can be increased by, for example, 0.2, and the updated image similarity is 1.0, the category of the second matching product library product does not match the category of the product to be processed, the image similarity of the second matching product library product remains unchanged, and the updated image similarity is still 0.85. At this time, the most matching product library product should be the first matching product library product.

[0127] As an implementation mode, when there are multiple matching product library products that are determined to be fused and matched with the product to be processed based on the fusion similarity, the fusion similarity of each matching product library product is updated based on whether the brand and category of each matching product library product match the brand and category of the product to be processed, and based on the updated fusion similarity, a matching product library product that is fused and matched with the product to be processed is determined.

[0128] Since the fusion similarity threshold is used for screening, there may be multiple matching product library products that meet the threshold requirements. At this time, similar to the image similarity, the fusion similarity can be updated. For specific content, please refer to the description of updating the image similarity above, which will not be repeated here.

[0129] According to the embodiments of the present application, by respectively using the text-side information and picture-side information of the goods to be processed and the goods in the goods library, it is possible to quickly identify pairs of goods that are obviously matched and obviously mismatched, and by integrating the text-side information and picture-side information of the goods to be processed and the goods in the goods library, it is possible to accurately identify pairs of goods that are suspected to be matched, thereby improving the efficiency and accuracy of product identification and achieving a balance between time and effect.

[0130] Those skilled in the art will understand that Figure 4 The method shown is based on Figure 2 and Figure 3 The trained graph-text model shown in the figure is implemented. For details, please refer to the above Figure 2 and Figure 3 The description will not be repeated here.

[0131] Figure 5 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0132] like Figure 5 As shown, the electronic device may include one or more processors 502, a system control logic 508 connected to at least one of the processors 502, a system memory 504 connected to the system control logic 508, a non-volatile memory (NVM) 506 connected to the system control logic 508, and a communication interface 510 connected to the system control logic 508.

[0133] The processor 502 may include one or more single-core or multi-core processors. The processor 502 may include any combination of a general-purpose processor and a special-purpose processor (e.g., a graphics processor, an application processor, a baseband processor, etc.). In an embodiment of the present application, the processor 502 may be configured to execute the following Figure 2-4 The method shown.

[0134] In some embodiments, system control logic 508 may include any suitable interface controller to provide any suitable interface to at least one of processors 502 and / or any suitable device or component in communication with system control logic 508 .

[0135] In some embodiments, the system control logic 508 may include one or more memory controllers to provide an interface to the system memory 504. The system memory 504 may be used to load and store data and / or instructions. In some embodiments, the system memory 504 of the electronic device may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0136] NVM 506 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM 506 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (CompactDisc) drive, and a DVD (Digital Versatile Disc) drive.

[0137] NVM 506 may include a portion of storage resources installed on a device of an electronic device, or it may be accessible by the device but not necessarily a portion of the device. For example, NVM 506 may be accessed over a network via communication interface 510 .

[0138] In particular, system memory 504 and NVM 506 may include, respectively, a temporary copy and a permanent copy of instructions 520. Instructions 520 may include instructions that, when executed by at least one of processors 502, cause the electronic device to implement the following: Figure 2-4 In some embodiments, instructions 520, hardware, firmware, and / or software components thereof may additionally / alternatively be placed in system control logic 508, communication interface 510, and / or processor 502.

[0139] The communication interface 510 may include a transceiver for providing a radio interface for the electronic device to communicate with any other suitable device (e.g., a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the communication interface 510 may be integrated with other components of the electronic device. For example, the communication interface 510 may be integrated with at least one of the processor 502, the system memory 504, the NVM 506, and a firmware device (not shown) having instructions. When at least one of the processors 502 executes the instructions, the electronic device implements Figure 2-4 One or more of the various embodiments shown.

[0140] The communication interface 510 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the communication interface 510 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0141] In one embodiment, at least one of the processors 502 may be packaged together with one or more controllers for the system control logic 508 to form a system in package (SiP). In one embodiment, at least one of the processors 502 may be integrated on the same die with one or more controllers for the system control logic 508 to form a system on a chip (SoC).

[0142] The electronic device may further include: an input / output (I / O) device 512 connected to the system control logic 508. The I / O device 512 may include a user interface to enable a user to interact with the electronic device; the design of the peripheral component interface enables the peripheral component to interact with the electronic device. In some embodiments, the electronic device also includes a sensor for determining at least one of an environmental condition and location information related to the electronic device.

[0143] In some embodiments, I / O device 512 may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

[0144] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0145] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0146] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a system for processing instructions including a processor includes any system having a processor such as a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0147] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0148] According to an embodiment of the present application, a computer-readable storage medium is also proposed. The computer-readable storage medium stores one or more programs. The one or more programs can be executed by one or more processors to implement the aforementioned method.

[0149] According to one embodiment of the present application, a computer program product is also provided, and the aforementioned method is implemented when the computer program is executed by a processor.

[0150] The illustrative embodiments of the present application include, but are not limited to, a graphic model training method, a commodity recognition method, a device, and a medium.

[0151] The various aspects of the illustrative embodiments will be described using terms commonly used by those skilled in the art to convey the essence of their work to other persons skilled in the art. However, it is apparent to those skilled in the art that some alternative embodiments may be implemented using the features described in part. For purposes of explanation, specific numbers and configurations are set forth to provide a more thorough understanding of the illustrative embodiments. However, it is apparent to those skilled in the art that alternative embodiments may be implemented without the specific details. In some other cases, some well-known features are omitted or simplified herein to avoid obscuring the illustrative embodiments of the present application.

[0152] In addition, various operations will be described as multiple separate operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as implying that these operations must rely on the order of description, and many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can also be rearranged. When the described operations are completed, the process can be terminated, but can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0153] References in the specification to "an example", "an example", "an embodiment", "an implementation", etc. indicate that the described embodiment may include a particular feature, structure, or property, but each embodiment may or may not necessarily include the particular feature, structure, or property. Moreover, these phrases are not necessarily directed to the same embodiment. In addition, when particular features are described in conjunction with a specific embodiment, the knowledge of those skilled in the art can affect the combination of these features with other embodiments, whether or not these embodiments are explicitly described.

[0154] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A and / or B" means "(A), (B), or (A and B)."

[0155] As used herein, the term "module" may refer to, be part of, or include: memory (shared, dedicated, or group) for running one or more software or firmware programs, application-specific integrated circuits (ASICs), electronic circuits and / or processors (shared, dedicated, or group), combinational logic circuits, and / or other suitable components that provide the described functionality.

[0156] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not necessary. Instead, in some embodiments, these features may be described in a manner and / or order different from that shown in the illustrative drawings. In addition, the structural or method features included in a specific figure do not mean that all embodiments need to include such features. In some embodiments, these features may not be included or may be combined with other features.

[0157] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0158] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present application.

[0159] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0160] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

Claims

1. A method for training a graphic model, used in electronic equipment, characterized in that: The method comprises: Acquire training data, wherein the training data includes text and images related to a plurality of commodities; For each product, extract the named entity information of the product from the text related to the product, and extract the image information of the product from the image related to the product; Using the named entity information and the image information of the multiple commodities to train a pre-built model to obtain the trained image-text model, The pre-built model includes a first model, a second model and a third model, the named entity information of two of the multiple commodities is input into the first model to calculate the text similarity of the two commodities, the image information of two of the multiple commodities is input into the second model to calculate the image similarity of the two commodities, and the named entity information and the image information of two of the multiple commodities are input into the third model to calculate the fusion similarity of the two commodities, and the third model fuses the named entity information and the image information of the two commodities.

2. The method according to claim 1, characterized in that The named entity information includes one or more of a brand, a series, and a category of a product, and the image information includes a cropped image corresponding to a product detection frame.

3. The method according to claim 1, characterized in that The first model is a text Transformer model, the second model is an image Transformer model, and the third model is a fusion Transformer model.

4. A commodity identification method, used in electronic equipment, characterized in that: The method comprises: Receiving data to be processed and commodity library data, wherein the data to be processed includes text and pictures related to the commodity to be processed, and the commodity library data includes text and pictures related to commodities in multiple commodity libraries; The data to be processed and the commodity library data are used as inputs of the graphic model, and a matching result between the commodity to be processed and at least a part of the commodity library commodities in the plurality of commodity library commodities is outputted. The graphic model is trained according to any one of the methods in claims 1-3.

5. The method according to claim 4, characterized in that The method further comprises: For the commodity to be processed, the named entity information of the commodity to be processed is extracted from the text related to the commodity to be processed, and the image information of the commodity to be processed is extracted from the image related to the commodity to be processed; based on the named entity information and the image information of the commodity to be processed, at least a part of the commodity library commodities are recalled from the multiple commodity library commodities as candidate commodity library commodities, wherein each candidate commodity library commodity has the named entity information and the image information of the candidate commodity library commodity.

6. The method according to claim 5, characterized in that The method further comprises: Filtering the candidate product library products recalled based on the named entity information of the product to be processed, wherein the candidate product library products that do not match the brand, series and category of the product to be processed are removed; Inputting the named entity information of the to-be-processed product and each of the filtered candidate product library products into the first model to calculate the text similarity of the two products; Based on the text similarity, a matching product library product that matches the product text to be processed is determined, and an unmatched product library product that does not match the product text to be processed is excluded.

7. The method according to claim 6, characterized in that The method further comprises: Filtering the candidate product library products recalled based on the image information of the product to be processed, wherein the named entity information of the product and the candidate product library products are identified from the image information of the product and the candidate product library products respectively, and removing the candidate product library products that do not match the brand, series and category of the product to be processed; Inputting the image information of the product to be processed and each of the filtered candidate product library products into the second model to calculate the image similarity of the two products; Based on the image similarity, a matching product library product that matches the image of the product to be processed is determined, and an unmatched product library product that does not match the image of the product to be processed is excluded.

8. The method according to claim 7, characterized in that The method further comprises: For the to-be-processed product for which no matching product in the product library is determined, inputting the named entity information and the image information of the to-be-processed product and each of the excluded candidate product library products into the third model to calculate the fusion similarity of the two products; Based on the fusion similarity, a matching product library product that is fused and matched with the product to be processed is determined, and an unmatched product library product that is not fused and matched with the product to be processed is excluded.

9. The method according to claim 8, characterized in that In a case where there are multiple matching product library products that match the text of the product to be processed determined based on the text similarity, the image information of the product to be processed and each matching product library product is input into the second model to calculate the image similarity of the two products, and based on the text similarity and the image similarity, a matching product library product that matches the text of the product to be processed is determined.

10. The method according to claim 9, characterized in that In the case that there are multiple matching product library products that are matched or fused-matched with the image of the product to be processed based on the image similarity or the fusion similarity, the image similarity or the fusion similarity of each matching product library product is updated based on whether the brand and category of each matching product library product matches the brand and category of the product to be processed, and based on the updated image similarity or the fusion similarity, a matching product library product that is matched or fused-matched with the image of the product to be processed is determined.

11. A graphic model training device, characterized in that: The device comprises: an acquisition module configured to acquire training data, wherein the training data includes texts and pictures related to a plurality of commodities; an extraction module configured to extract, for each commodity, named entity information of the commodity from the text related to the commodity, and extract picture information of the commodity from the pictures related to the commodity; A training module is configured to train a pre-built model using the named entity information and the image information of the plurality of commodities to obtain the trained image-text model, The pre-built model includes a first model, a second model and a third model, the named entity information of two of the multiple commodities is input into the first model to calculate the text similarity of the two commodities, the image information of two of the multiple commodities is input into the second model to calculate the image similarity of the two commodities, and the named entity information and the image information of two of the multiple commodities are input into the third model to calculate the fusion similarity of the two commodities, and the third model fuses the named entity information and the image information of the two commodities.

12. A commodity identification device, characterized in that: The device comprises: A receiving module, configured to receive data to be processed and commodity library data, wherein the data to be processed includes text and pictures related to the commodity to be processed, and the commodity library data includes text and pictures related to commodities in multiple commodity libraries; The recognition module is configured to use the data to be processed and the product library data as inputs of a graphic model, and output matching results between the product to be processed and at least a portion of the product library products in the multiple product library products, wherein the graphic model is trained according to the method described in any one of claims 1 to 3.

13. An electronic device, characterized in that: The electronic device comprises a memory storing computer executable instructions and a processor; when the computer executable instructions are executed by the processor, the electronic device implements the method according to any one of claims 1 to 3, or implements the method according to any one of claims 4 to 10.

14. A computer readable medium, characterized in that The computer-readable medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 3, or to implement the method according to any one of claims 4 to 10.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented, or the method according to any one of claims 4 to 10 is implemented.