Optimization method and device for improving cross-domain search effect

Abstract

The application provides an optimization method and device for improving cross-domain retrieval effect, and belongs to the technical field of data processing systems under the internet industry, such as a system based on a specific computing model or a data processing system or method technology field specially used for business. The feature coding model provided by the application can cross the difference in the visual dimension between e-commerce commodity illustrations and appearance patent illustrations. The system first extracts feature vectors of text and images by using natural language processing technology, computer vision technology and a deep learning model, then realizes cross-modal mapping of images and text descriptions and cross-domain mapping between e-commerce commodity illustrations and appearance patent illustrations by using a contrast learning method, so that the similarity between different image forms can be captured, common features between e-commerce commodity illustrations and appearance patent illustrations can be effectively identified, and the accuracy of cross-domain appearance patent retrieval of an appearance retrieval recall system can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing systems under the Internet industry, such as systems based on specific computing models or data processing systems or method technologies dedicated to business, and particularly relates to an optimization method and device for improving cross-domain retrieval effect. BACKGROUND

[0002] In the retrieval process of appearance patents, accurately identifying and matching cross-domain images (such as e-commerce commodity illustrations and appearance patent illustrations) is an important challenge. Traditional appearance patent retrieval systems usually rely on text descriptions and simple image matching techniques, which show obvious limitations in handling different forms and styles of images. SUMMARY

[0003] The present application provides an optimization method and device for improving cross-domain retrieval effect. Through the feature encoding model provided by the present application, the differences in visual dimensions between e-commerce commodity illustrations and appearance patent illustrations can be bridged, thereby capturing the similarities between different image forms, effectively identifying the common features between e-commerce commodity illustrations and appearance patent illustrations, and facilitating the improvement of the accuracy of the cross-domain appearance patent retrieval of the appearance retrieval recall system.

[0004] In a first aspect, the application provides an optimization method for improving cross-domain search effect, applied to an appearance patent search recall system of an e-commerce platform, the method comprising: processing an appearance patent database of a target area based on prompt word engineering to obtain a patent description dataset, the patent description dataset containing normalized descriptions of legends and text titles of a plurality of appearance patents in the appearance patent database, and the plurality of appearance patents containing appearance patents under a plurality of Locarno categories; using a pre-trained first CLIP series model to encode the patent description dataset to obtain an encoded result dataset of the patent description dataset, the first CLIP series model including a text encoder and a visual encoder; and using the encoded result dataset, an appearance patent dataset, and a contrastive learning loss function to perform first training on the first CLIP series model with activated full parameters to obtain a second CLIP series model; obtaining positive sample data containing e-commerce commodity legends and infringing appearance patent legends; and constructing a plurality of three tuple dataset sets with positive and negative sample data ratio differentiation according to the positive sample data; using the plurality of three tuple dataset sets and a TripletLoss loss function to perform second training on the second CLIP series model to obtain a third CLIP series model, the TripletLoss loss function being used to shorten the spatial distance between the infringing appearance patent legends and the e-commerce commodity legends in the positive sample data, and to lengthen the spatial distance between the infringing appearance patent legends and non-infringing e-commerce commodity legends in the negative sample data; calling the third CLIP series model to process user-uploaded to-be-searched commodity image-text data to obtain to-be-searched commodity image-text encoding data; and using the to-be-searched commodity image-text encoding data to search the appearance patent database to obtain a search result.

[0005] In a second aspect, the application provides an optimization device for improving cross-domain retrieval effect, applied to an appearance patent retrieval recall system of an e-commerce platform. The device comprises: a creating unit configured to process a target area appearance patent database based on a prompt word to obtain a patent description dataset, wherein the patent description dataset comprises normalized descriptions of legends and text titles of a plurality of appearance patents in the appearance patent database, and the plurality of appearance patents comprise appearance patents under a plurality of Locarno categories; a first training unit configured to encode the patent description dataset using a pre-trained first CLIP series model to obtain an encoded result dataset of the patent description dataset, wherein the first CLIP series model comprises a text encoder and a visual encoder; and perform first training on the first CLIP series model with activation full parameters using the encoded result dataset, an appearance patent dataset, and a contrastive learning loss function to obtain a second CLIP series model; an obtaining unit configured to obtain positive sample data comprising e-commerce commodity legends and infringing appearance patent legends; and construct a plurality of three tuple datasets with positive and negative sample data matching differences according to the positive sample data; a second training unit configured to perform second training on the second CLIP series model using the plurality of three tuple datasets and a TripletLoss loss function to obtain a third CLIP series model, wherein the TripletLoss loss is used to shorten the spatial distance between the infringing appearance patent legends and the e-commerce commodity legends in the positive sample data, and to lengthen the spatial distance between the infringing appearance patent legends and non-infringing e-commerce commodity legends in the negative sample data; a calling unit configured to call the third CLIP series model to process user-uploaded to-be-retrieved commodity image-text data to obtain to-be-retrieved commodity image-text encoding data; and use the to-be-retrieved commodity image-text encoding data to retrieve the appearance patent database to obtain a retrieval result.

[0006] As can be seen, in the embodiments of the application, the feature encoding model provided by the application can bridge the differences in visual dimensions between e-commerce commodity legends and appearance patent legends. The system first extracts feature vectors of text and images using natural language processing technology, computer vision technology, and a deep learning model, and then realizes cross-modal mapping of images and text descriptions and cross-domain mapping between e-commerce commodity legends and appearance patent legends using a contrastive learning method, so as to capture similarities between different image forms, effectively identify common features between e-commerce commodity legends and appearance patent legends, and improve the accuracy of cross-domain appearance patent retrieval by the appearance retrieval recall system. BRIEF DESCRIPTION OF DRAWINGS

[0007] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings described in the following embodiments are only some of the embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on these drawings also belong to the protection scope of the present application.

[0008] Figure 1 A structural schematic diagram of an appearance patent retrieval recall system provided by the embodiments of the present application is shown in FIG. 1.

[0009] Figure 2 A flowchart of an optimization method for improving cross-domain retrieval effect provided by the embodiments of the present application is shown in FIG. 2.

[0010] Figure 3 A positive sample data creation diagram provided by the embodiments of the present application is shown in FIG. 3.

[0011] Figure 4 A display interface diagram of first and second positive sample data provided by the embodiments of the present application is shown in FIG. 4.

[0012] Figure 5 A structural schematic diagram of a server provided by the embodiments of the present application is shown in FIG. 5.

[0013] Figure 6 A functional unit structure block diagram of an optimization device for improving cross-domain retrieval effect provided by the embodiments of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0014] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings described in the following embodiments are only some of the embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on these drawings also belong to the protection scope of the present application.

[0015] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, product or device.

[0016] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. It is expressly understood that any of the embodiments described herein can be incorporated in a combination of embodiments.

[0017] In the embodiments of the present application, “and / or” describes the association relationship of the associated objects, which means that there can be three kinds of relationships. For example, A and / or B can represent the following three cases: A exists alone; A and B exist simultaneously; B exists alone. Wherein, A and B can be singular or plural.

[0018] In the embodiments of the present application, the symbol“ / ” can represent that the associated objects before and after the symbol are in an“or” relationship. In addition, the symbol“ / ” can also represent the division sign, that is, performing division operation. For example, A / B can represent A divided by B.

[0019] In the embodiments of the present application, “at least one” or similar expressions mean any combination of these items, including any combination of single item or multiple items, means one or more, and multiple means two or more. For example, at least one of a, b or c can represent the following seven cases: a, b, c, a and b, a and c, b and c, a, b and c. Wherein, each of a, b and c can be an element or a set containing one or more elements.

[0020] In the embodiments of the present application, “equal to” can be combined with greater than, which is applicable to the technical solutions adopted when greater than, or combined with less than, which is applicable to the technical solutions adopted when less than. When equal to is combined with greater than, it is not combined with less than; when equal to is combined with less than, it is not combined with greater than.

[0021] In the e-commerce platform application scenario, there is a significant difference between the e-commerce graphic information of the goods uploaded by the merchant user to the e-commerce platform and the patent graphic information in the appearance patent database. The e-commerce graphic information usually contains rich details, colors, shadows, decorations, accessories, detachable components and other feature information, while the patent graphic information tends to be simplified lines, overall contours, general functional purposes and other feature information. Such differences in feature information make it difficult for traditional image retrieval techniques to effectively match between different forms of images. In addition, due to the scarcity of appearance patent graphic data, the general multi-modal model has poor discrimination ability for appearance patent graphics, and it is difficult to realize unified representation of appearance image information and text information.

[0022] To solve these problems, the application provides an optimization method and device for improving cross-domain retrieval effect. The feature coding model provided by the application can cross the difference in visual dimension between e-commerce commodity illustrations and appearance patent illustrations, thereby capturing the similarity between different image forms, effectively identifying the common features between e-commerce commodity illustrations and appearance patent illustrations, and facilitating to improve the accuracy of cross-domain appearance patent retrieval of the appearance retrieval recall system.

[0023] The technical solutions of the application and how the technical solutions solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the application will be described below with reference to the drawings.

[0024] The embodiment of the application provides an e-commerce platform, which comprises an appearance patent retrieval recall system, and the appearance patent retrieval recall system is used to execute each step instruction in the method embodiment of the optimization method for improving cross-domain retrieval effect.

[0025] Please refer to Figure 1 , Figure 1 The structure diagram of the appearance patent retrieval recall system provided by the embodiment of the application is shown in Figure 1 The appearance patent retrieval recall system 1 comprises a server 10 and a terminal device 20, and the server 10 is in communication connection with the terminal device 20.

[0026] The server 10 can specifically comprise a server responsible for data processing and applied to the network platform side, which can realize data transmission, data processing and other functions. The server can be a physical server, a server cluster composed of multiple physical servers or a distributed system. In the embodiment, the number of servers is not specifically limited. Alternatively, the server can be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0027] When specifically implemented, please refer to Figure 5 , Figure 5 The structure diagram of the server provided by the embodiment of the application is shown in Figure 5As shown, the server 10 comprises a processor 1001, a memory 1003, a communication interface 1002, and one or more programs 10031 stored in the memory 1003 and configured to be executed by the processor 1001, the programs comprising instructions for performing the methods described in the embodiments above.

[0028] The terminal device 20 can specifically include a front-end device applied to a user side and capable of realizing data acquisition, data transmission and the like, and can be a mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA), a tablet computer (PAD), AR glasses, a user equipment (UE), a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing device connected to a wireless modem, a mobile station (MS), a mobile terminal (MT) and the like. Alternatively, the terminal device 20 can also be a software application capable of running in the electronic device. For example, it can be an APP running on a mobile phone and the like.

[0029] Referring to Figure 2 , Figure 2 A flowchart of an optimization method for improving cross-domain retrieval effect provided by an embodiment of the present application is provided, and the method is applied to an appearance patent retrieval recall system of an e-commerce platform, as shown in Figure 2 The method comprises the following steps S201-S208:

[0030] In step S201, a patent description data set is obtained by engineering processing a target area based on a prompt word.

[0031] The patent description data set contains normalized descriptions of legends and text titles of a plurality of appearance patents in the appearance patent database, and the plurality of appearance patents contain appearance patents under a plurality of Locarno categories.

[0032] In some embodiments, the appearance patent database based on the prompt word engineering processing target region obtains a patent description dataset, including: deploying a visual large model using a large model inference framework; constructing a plurality of candidate thought chains COTs, and calling each COT and the visual large model to perform a patent description task on a preset appearance patent dataset to obtain a plurality of patent description test datasets corresponding to the plurality of COTs, the preset appearance patent dataset containing part of original appearance patent data in the appearance patent database of the target region, and the patent description dataset containing appearance patent illustrations and appearance patent text descriptions; evaluating the plurality of patent description test datasets in a semi-automatic evaluation manner of manual sampling review and Rouge-L index calculation, and selecting a target COT with the best comprehensive performance; and processing the appearance patent database of the target region using the target COT to obtain the patent description dataset.

[0033] The large model inference framework includes, but is not limited to, VLM, LMDeploy, etc. The large model inference framework is a collection of software tools for running and managing visual large models, which can configure and start the trained visual large model on a specific hardware environment (such as a server, a computer, etc.), so that the model can receive input data such as image visual information and perform inference calculation, and output the response processing result.

[0034] The thought chain COT is a logical chain of step-by-step reasoning, and by constructing a plurality of different candidate COTs, the visual large model is tried to be guided to process appearance patent data from different reasoning paths or angles. By calling each candidate COT and the visual large model to process the preset appearance patent dataset to obtain the corresponding patent description test dataset, that is, each candidate COT processes to obtain a patent description test dataset, and then each patent description test dataset is evaluated in a semi-automatic evaluation manner of manual sampling review and Rouge-L index calculation, and the performance of each COT corresponding to the patent description test dataset is considered comprehensively, and a COT that performs well in each evaluation index is selected as the target COT.

[0035] Next, the target COT is used to process the appearance patent database of the target region to obtain a patent description dataset of the appearance patent of the target region, and the patent description dataset contains the standardized description of the illustrations and text titles of a plurality of appearance patents in the appearance patent database of the target region. The appearance patent database of the target region collects millions of patent pictures and title data under different Locarno catalogs, and the target COT processes the appearance patent database of the target region to generate corresponding high-quality patent descriptions.

[0036] In some embodiments, the COT needs to explicitly include the task content, the target, and the matters needing attention, etc.

[0037] For example, the task content of the first COT candidate is to observe the overall shape and main contour of the appearance patent figure, describe the basic shape characteristics (such as circle, square, etc.), then describe the main detail part (such as whether there is a protrusion, a depression, etc.), and finally summarize the overall design style. The target of the first COT is to generate an accurate and detailed appearance patent text description, highlighting the key features of the design. The matters needing attention of the first COT is to avoid using ambiguous or inaccurate words, and to ensure that the description is consistent with the figure.

[0038] The task content of the second COT candidate is to analyze the colors and materials of each part in the appearance patent figure first, then describe the relative position relationship between each part, then point out the possible advantages or characteristics of the design in function, and finally give the overall appearance evaluation. The target of the second COT is to comprehensively describe the appearance patent from the aspects of color, material, function, etc., to help understand the intention and value of the design. The matters needing attention of the second COT is that for the uncertain color and material description, speculative but reasonable language can be used.

[0039] It should be noted that the specific content of the above two COTs is only an example provided by the present application and does not limit the present application itself.

[0040] Step S202, using a pre-trained first CLIP series model to encode the patent description data set to obtain an encoded result data set of the patent description data set.

[0041] The first CLIP series model includes a text encoder and a visual encoder. The appearance patent text in the patent description data set is encoded by the text encoder, and the appearance patent figure in the patent description data set is encoded by the visual encoder, which facilitates subsequent model processing and analysis.

[0042] In some embodiments, encoding refers to converting text or figures into corresponding feature vectors.

[0043] Step S203, using the encoded result data set, the appearance patent data set, and the contrastive learning loss function to perform first training on the first CLIP series model with full activation to obtain a second CLIP series model.

[0044] In this step, the previously obtained encoding result dataset of the patent description dataset, the appearance patent dataset, are used to train the first CLIP series model in combination with the contrastive learning loss function. Activating all parameters means that all parameters of the model will be updated and optimized during the training process. Through training, the model can learn the association between the appearance patent description and the appearance patent illustration, thereby obtaining a second CLIP series model with better performance.

[0045] Step S204, obtaining positive sample data containing e-commerce commodity illustrations and infringing appearance patent illustrations.

[0046] Among them, the positive sample data obtained here refers to the positive sample data of each type of e-commerce product obtained by targeting all e-commerce commodities on the target regional e-commerce platform. For example, the products sold on the e-commerce platform include sofas, clothes, electronic products such as smart watches, mobile phones, earphones, etc. Then the positive sample data of sofas, clothes, smart watches, mobile phones, earphones, etc. will be obtained.

[0047] In some embodiments, the step of obtaining positive sample data containing e-commerce commodity illustrations and infringing appearance patent illustrations includes steps A1-A4 as follows:

[0048] Step A1, obtaining e-commerce commodity illustrations, commodity main bodies, and dispute handling records of target commodities on the target regional e-commerce platform.

[0049] Among them, the target commodity can be any type of e-commerce product on the target regional e-commerce platform. For each type of product on the e-commerce platform, its positive sample data is obtained according to the method shown in steps A1-A4.

[0050] Among them, the commodity main body is used to represent the relevant main body that may register an appearance design for the appearance design of the target commodity, and the dispute handling record refers to the record information of abnormal events of the target commodity or the commodity main body other than normal sales events.

[0051] For example, the target regional e-commerce platform is A platform in country A, the target commodity is a smart watch, the commodity main body may be a manufacturer other than the main body corresponding to the target product, and the dispute handling record may include records of abnormal events such as appearance infringement and appearance disputes with other brands that the smart watch has been involved in.

[0052] Step A2, if it is detected that the dispute handling record contains record information of an appearance infringement event, then first-type positive sample data is created according to the e-commerce commodity illustrations in the record information of the appearance infringement event and the cited appearance patent.

[0053] If the record information of the appearance infringement event is detected according to the obtained dispute handling record, the e-commerce commodity icon and the cited appearance patent icon in the record information are combined to obtain the first type of positive sample data.

[0054] In some embodiments, the e-commerce commodity icon and the cited appearance patent icon in the first type of positive sample data have a high similarity.

[0055] Step A3, querying the appearance patent database according to the commodity main body to determine whether there is a registered appearance patent.

[0056] Step A31, if it is determined that there is a registered appearance patent, creating a second type of positive sample data according to the registered appearance patent and the e-commerce commodity icon; and marking the first type of positive sample data and the second type of positive sample data as the positive sample data.

[0057] Step A32, if it is determined that there is no registered appearance patent, marking the first type of positive sample data as the positive sample data.

[0058] According to the commodity main body information, the appearance patent database is queried to determine whether the commodity main body has a registered appearance patent. Here, it mainly refers to the appearance patent related to the target product. If there is a registered appearance patent, a second type of positive sample data is created according to the registered appearance patent and the e-commerce commodity icon. The first type of positive sample data and the second type of positive sample data are marked as the positive sample data. If there is no registered appearance patent, the first type of positive sample data is directly used as the positive sample data. The positive sample data is an icon that has a strong similarity with the e-commerce commodity icon of the target product.

[0059] Step A4, if it is detected that the dispute handling record does not have record information of the appearance infringement event, querying the appearance patent database according to the commodity main body to determine whether there is a registered appearance patent.

[0060] Step A41, if it is determined that there is a registered appearance patent, creating a second type of positive sample data according to the registered appearance patent and the e-commerce commodity icon; and marking the second type of positive sample data as the positive sample data.

[0061] Step A42, if it is determined that there is no registered appearance patent, continuing to process the next commodity until a preset number of positive sample data are obtained.

[0062] If no record of design infringement is found in the dispute resolution records, the system directly queries the design patent database based on the product owner information to determine if the product owner has a registered design patent. If no registered design patent exists, the system continues processing the next product. If a registered design patent exists, the system retrieves registered design patents that have a high degree of similarity to the e-commerce product illustration of the target product. Based on the e-commerce product illustration and the registered design patent, a second type of positive sample data is created and marked as positive sample data.

[0063] As can be seen, in this embodiment, the appearance patents of the product are comprehensively queried and matched based on the multi-dimensional information of the product on the e-commerce platform, thereby creating positive sample data and improving the comprehensiveness and efficiency of creating positive sample data.

[0064] In some embodiments, creating a second type of positive sample data based on the registered design patent and the e-commerce product illustration includes: performing a preliminary similarity check on the design patent illustration of the registered design patent using the e-commerce product illustration, and obtaining the preliminary similarity check result; displaying multiple candidate design patent illustrations ranked first by the preliminary similarity check result on a terminal device; and creating a second type of positive sample data based on at least one design patent illustration determined by user manual screening and the e-commerce product illustration.

[0065] As can be seen, in this embodiment, by performing a similarity query on registered design patents and e-commerce product illustrations, it is possible to obtain registered design patent illustrations that are consistent with or similar to e-commerce products, thereby creating a second type of positive sample data.

[0066] In summary, e-commerce product illustrations refer to the illustrations of the target product, while infringing design patent illustrations refer to design patent illustrations that have a high degree of similarity to e-commerce product illustrations. This includes design patents cited in infringement dispute resolution records and registered design illustrations in the registered appearance of the product entity that have a high degree of similarity to e-commerce product illustrations.

[0067] For example, please refer to Figures 3-4 ,See Figure 3 , Figure 3 A schematic diagram illustrating the creation of positive sample data provided in the embodiments of this application, such as... Figure 3 As shown, the first display page 3 of the design patent search and recall system includes the target area and target e-commerce platform, as well as all product categories 31 on the target e-commerce platform. Selecting a target product from all product categories 31 displays the product information 32, including e-commerce product images of the target product, the relevant product entities, and dispute resolution records. The first display page 3 also includes a positive sample data acquisition control. Upon detecting a selection operation on the positive sample data acquisition control, the first display page 3 displays first-type and second-type positive sample data.Figure 4 The first type of positive sample data is an appearance pattern that is disputed with the appearance pattern of the e-commerce commodity pattern of the target product, and the second type of positive sample data is a registered trademark pattern in the registered appearance patent of the e-commerce subject that has high similarity with the e-commerce commodity pattern of the target product. The user can select at least one sample data with high actual similarity from the first type of positive sample data and the second type of positive sample data as the positive sample data of the target product.

[0068] The target product can be selected by the user from all product categories, or can be selected by the system for all or part of the products in turn. When a product is processed, the next product is continued to acquire positive sample data, and the process is repeated until a preset number of positive sample data is acquired.

[0069] Step S205, constructing a plurality of triplet data sets with different ratios of positive and negative sample data according to the positive sample data.

[0070] In some embodiments, the plurality of triplet data sets with different ratios of positive and negative sample data are constructed according to the positive sample data, including: matching at least one negative sample data to the positive sample data in a random sampling manner; and splicing each negative sample data and the positive sample data to obtain the plurality of triplet data sets with different ratios of positive and negative sample data.

[0071] The negative sample data refers to an appearance pattern that does not constitute an infringement relationship with the e-commerce commodity pattern of the target product or has low similarity. For each positive sample data, at least one negative sample is randomly selected from the negative sample data set to match it. The random sampling manner can ensure the randomness of the sample and avoid overfitting of the model to specific negative samples.

[0072] In some embodiments, the triplet data set contains a plurality of triplet data, and the expression of a single triplet data can be, for example: (e-commerce commodity pattern-A platform infringement appearance pattern-A platform non-infringement appearance pattern).

[0073] The establishment of the negative sample data set is as follows: acquiring a registered appearance patent in the registered appearance patent set of the product subject that has a similarity lower than a preset similarity with the e-commerce commodity pattern of the target product, and marking the e-commerce commodity pattern and the registered appearance patent as negative sample data.

[0074] Step S206, using the plurality of triplet data sets and the TripletLoss loss function to perform a second training on the second CLIP series model to obtain a third CLIP series model.

[0075] The TripletLoss loss is used to reduce the spatial distance between the infringing appearance patent image and the e-commerce commodity image in the positive sample data, and to increase the spatial distance between the infringing appearance patent image and the non-infringing e-commerce commodity image in the negative sample data.

[0076] In some embodiments, the second training of the second CLIP series model by using the plurality of triplet data sets and the TripletLoss loss function to obtain a third CLIP series model comprises the following steps B1-B3:

[0077] Step B1, freeze the text encoder of the second CLIP series model, only activate the LayerNorm layer in the visual encoder of the second CLIP series model, and encode each triplet data in the plurality of triplet data sets to obtain a triplet data set encoding result.

[0078] Step B2, calculate a loss value by using the triplet data set encoding result and the TripletLoss loss function.

[0079] Step B3, adjust the network parameters of the LayerNorm layer by using the back propagation algorithm through the loss value to perform the second training of the second CLIP series model to obtain a third CLIP series model.

[0080] In some embodiments, the calculation formula of the TripletLoss loss function can be, for example:

[0081] L=max(d(v(A), v(P)) d(v(A), v(N))+α,0)

[0082] v(·)=m·x+n·t

[0083] wherein v(A) is the encoding vector of the target product, v(P) is the encoding vector of the infringing appearance, v(N) is the encoding vector of the non-infringing appearance, d represents the distance between two encoding vectors, α is a preset boundary value, L is the loss value of the triplet data, the smaller the loss value, the more the model learned relationship conforms to the expectation, A represents the patent description data set of the target product, P represents the patent description data set of the infringing appearance, N represents the patent description data set of the non-infringing appearance, v(·) represents the vector encoding function of the patent description data set, x is the image encoding feature, t is the text encoding feature, m is the weight parameter of the image encoding feature, n is the weight parameter of the text encoding feature, and m+n=1.

[0084] In some embodiments, the weight parameters of the image encoding features and the text encoding features in the TripletLoss loss function are designed according to the product categories of the e-commerce product illustrations.

[0085] Taking a sofa as an example, its appearance characteristics may be more important in infringement judgment, such as the shape, material texture, etc. of the sofa. Therefore, in the TripletLoss loss function, the feature weight of the image dimension can be set relatively high. While the text description (such as the function, size, etc. of the sofa) has relatively less impact on appearance infringement judgment, so the text weight can remain basically the same as other product categories. For example, when calculating the loss, the weight of the image features is 0.8, and the weight of the text features is 0.2. When calculating the loss of the triplet data of the sofa, more emphasis will be placed on the distance calculation between the image features, so as to better capture the similarities and differences in the appearance of the sofa.

[0086] Taking a sofa as an example, its appearance characteristics may be more important in infringement judgment, such as the shape, material texture, etc. of the sofa. Therefore, in the TripletLoss loss function, the feature weight of the image dimension can be set relatively high. While the text description (such as the function, size, etc. of the sofa) has relatively less impact on appearance infringement judgment, so the text weight can remain basically the same as other product categories. For example, when calculating the loss, the weight of the image features is 0.8, and the weight of the text features is 0.2. When calculating the loss of the triplet data of the sofa, more emphasis will be placed on the distance calculation between the image features, so as to better capture the similarities and differences in the appearance of the sofa.

[0087] Step S207, calling the third CLIP series model to process the user uploaded product image-text data to be searched to obtain product image-text encoding data to be searched.

[0088] After obtaining the third CLIP series model, input the user uploaded product image-text data to be searched into the model. The model will process the image-text data and encode it into a feature vector to obtain product image-text encoding data to be searched. These encoding data can represent the features of the product to be searched and be used for subsequent search tasks.

[0089] Step S208, searching the appearance patent database using the product image-text encoding data to be searched to obtain a search result.

[0090] Compare the product image-text encoding data to be searched with the patent image-text encoding data in the appearance patent database to calculate the similarity between them. According to the similarity ranking, obtain a list of appearance patents most similar to the product to be searched as a search result.

[0091] It can be seen that in the embodiments of the present application, the feature coding model provided by the present application can cross the difference in visual dimension between e-commerce commodity illustrations and appearance patent illustrations. The system first extracts feature vectors of text and images using natural language processing technology, computer vision technology and deep learning model, and then realizes cross-modal mapping of images and text descriptions and cross-domain mapping between e-commerce commodity illustrations and appearance patent illustrations by a contrast learning method, so as to capture the similarity between different image forms, effectively identify the common features between e-commerce commodity illustrations and appearance patent illustrations, and facilitate to improve the accuracy of cross-domain appearance patent retrieval of the appearance retrieval recall system.

[0092] The above describes the scheme of the embodiments of the present application mainly from the perspective of the execution process of the method. It can be understood that the server includes hardware structures and / or software modules corresponding to the execution of each function in order to implement the above functions. Those skilled in the art should easily realize that, in combination with the unit and algorithm steps of each example described in the embodiments provided herein, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0093] The embodiments of the present application can divide the server into functional units according to the above method examples. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated unit can be realized in the form of hardware or in the form of a software program module. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical functional division, and actual implementation can have another division method.

[0094] In the case of using integrated units, please refer to Figure 6 , Figure 6 A functional unit structure block diagram of an optimization device for improving cross-domain retrieval effect provided by the embodiments of the present application, the optimization device for improving cross-domain retrieval effect is applied to an appearance patent retrieval recall system of an e-commerce platform, and the optimization device for improving cross-domain retrieval effect 6 includes:

[0095] A creating unit 601 is configured to obtain a patent description data set by engineering processing an appearance patent database of a target area based on a prompt word, wherein the patent description data set includes standardized descriptions of illustrations and text titles of a plurality of appearance patents in the appearance patent database, and the plurality of appearance patents include appearance patents under a plurality of Locarno categories.

[0096] The first training unit 602 is configured to encode the patent description dataset by using a pre-trained first CLIP series model to obtain an encoded result dataset of the patent description dataset, the first CLIP series model comprising a text encoder and a visual encoder; and perform first training on the first CLIP series model with activation parameters by using the encoded result dataset, an appearance patent dataset, and a contrastive learning loss function to obtain a second CLIP series model.

[0097] The acquisition unit 603 is configured to acquire positive sample data containing e-commerce commodity images and infringing appearance patent images; and construct a plurality of three tuple data sets with positive and negative sample data matching differences according to the positive sample data.

[0098] The second training unit 604 is configured to perform second training on the second CLIP series model by using the plurality of three tuple data sets and a TripletLoss loss function to obtain a third CLIP series model, the TripletLoss loss being used to shorten the spatial distance between the infringing appearance patent images and the e-commerce commodity images in the positive sample data, and to lengthen the spatial distance between the infringing appearance patent images and the non-infringing e-commerce commodity images in the negative sample data.

[0099] The calling unit 605 is configured to call the third CLIP series model to process user-uploaded to-be-retrieved commodity image-text data to obtain to-be-retrieved commodity image-text encoded data; and use the to-be-retrieved commodity image-text encoded data to retrieve the appearance patent database to obtain a retrieval result.

[0100] As can be seen, in the embodiments of the present application, the feature encoding model provided by the present application can cross the difference in the visual dimension between e-commerce commodity images and appearance patent images. The system first extracts feature vectors of text and images by using natural language processing technology, computer vision technology and a deep learning model, and then realizes cross-modal mapping of images and text descriptions and cross-domain mapping between e-commerce commodity images and appearance patent images by using a contrastive learning method, so as to capture the similarity between different image forms, effectively identify the common features between e-commerce commodity images and appearance patent images, and facilitate to improve the accuracy of cross-domain appearance patent retrieval by the appearance retrieval recall system.

[0101] In some embodiments, in the aspect of obtaining the patent description dataset based on the prompt word engineering processing of the appearance of the target region, the creating unit 601 is specifically configured to: deploy a visual large model by using a large model inference framework; construct a plurality of candidate thinking chain prompts COTs, and call each COT and the visual large model to perform a look-at-picture-speaking task on a preset appearance patent dataset to obtain a plurality of patent description test datasets corresponding to the plurality of COTs, wherein the preset appearance patent dataset contains part of original appearance patent data in the appearance patent database of the target region, and the patent description dataset contains appearance patent illustrations and appearance patent text descriptions; and evaluate the plurality of patent description test datasets by using a semi-automatic evaluation method of artificial sampling review and Rouge-L index calculation, and select a target COT with the best comprehensive performance; and process the appearance patent database of the target region by using the target COT to obtain the patent description dataset.

[0102] In some embodiments, in the aspect of obtaining the positive sample data containing e-commerce commodity illustrations and infringing appearance patent illustrations, the obtaining unit 603 is specifically configured to: obtain an e-commerce commodity illustration, a commodity main body, and a dispute handling record of a target commodity of an e-commerce platform in the target region, wherein the commodity main body is used to represent a related subject who is likely to register an appearance design for the target commodity, and the dispute handling record is record information of an abnormal event of the target commodity or the commodity main body other than a normal sales event; if it is detected that the dispute handling record has record information of an appearance infringement event, then first-class positive sample data is created according to the e-commerce commodity illustration and the cited appearance patent in the record information of the appearance infringement event; and the appearance patent database is queried according to the commodity main body to determine whether there is a registered appearance patent; if it is determined that there is the registered appearance patent, then second-class positive sample data is created according to the registered appearance patent and the e-commerce commodity illustration; and the first-class positive sample data and the second-class positive sample data are marked as the positive sample data; if it is determined that there is no registered appearance patent, then the first-class positive sample data is marked as the positive sample data; if it is detected that the dispute handling record does not have record information of an appearance infringement event, then the appearance patent database is queried according to the commodity main body to determine whether there is a registered appearance patent; if it is determined that there is the registered appearance patent, then second-class positive sample data is created according to the registered appearance patent and the e-commerce commodity illustration; and the second-class positive sample data is marked as the positive sample data; if it is determined that there is no registered appearance patent, then the next commodity is processed until a preset number of positive sample data is obtained.

[0103] In some embodiments, in the aspect of constructing the plurality of triplet data sets with positive-negative sample data matching difference, the obtaining unit 603 is specifically configured to: match at least one negative sample data for the positive sample data in a random sampling manner; and splice each negative sample data and the positive sample data to obtain the plurality of triplet data sets with positive-negative sample data matching difference.

[0104] In some embodiments, in the aspect of training the second CLIP series model based on the plurality of triplet data sets and the TripletLoss loss function to obtain a third CLIP series model, the second training unit 604 is specifically configured to: freeze the text encoder of the second CLIP series model, activate only the LayerNorm layer in the visual encoder of the second CLIP series model, encode each triplet data in the plurality of triplet data sets respectively to obtain triplet data set encoding results; calculate a loss value based on the triplet data set encoding results and the TripletLoss loss function, and adjust the network parameters of the LayerNorm layer based on the loss value by using the back propagation algorithm, so as to train the second CLIP series model based on the loss value to obtain the third CLIP series model.

[0105] The embodiments of the present application provide a computer readable storage medium, which stores computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the method described in any possible embodiment.

[0106] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, some steps can be performed in other sequences or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0107] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0108] In several embodiments provided in the present application, it should be understood that the disclosed apparatus can be implemented in other manners. For example, the division of the apparatus embodiments described above is merely illustrative, and the division of the units can be different, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0109] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0110] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0111] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable memory. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or all or part of the technical solutions can be embodied in the form of a software product, which is stored in a memory and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the above-mentioned method of each embodiment of the present application. The aforementioned memory includes: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0112] Those of ordinary skill in the art can understand that all or part of the steps of the various methods of the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer readable memory, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0113] The above has carried out the detailed introduction to the embodiment of the application, the principle and implementation mode of the application have been described in this article by applying specific examples, the above embodiment explanation is only for helping understanding the method of the application and its core idea; at the same time, for the general technical personnel in the art, according to the idea of the application, the specific implementation mode and application range will have the change, and the above-mentioned, the content of the specification should not be understood as the limitation of the application.

Claims

1. An optimization method for improving cross-domain retrieval performance, characterized in that, A design patent retrieval and recall system applied to e-commerce platforms, the method comprising: Based on the prompt word engineering processing of the target area of ​​the appearance patent database, a patent description dataset is obtained. The patent description dataset contains standardized descriptions of legends and text titles of multiple appearance patents in the appearance patent database, and the multiple appearance patents include appearance patents under multiple Locarno categories. The patent description dataset is encoded using a pre-trained first CLIP series model to obtain an encoded result dataset of the patent description dataset. The first CLIP series model includes a text encoder and a visual encoder. The first CLIP series model with all parameters activated is trained using the encoded result dataset, the appearance patent dataset, and a contrastive learning loss function to obtain a second CLIP series model. Obtain positive sample data containing e-commerce product illustrations and infringing design patent illustrations; and construct multiple triplet datasets with differentiated positive and negative sample data ratios based on the positive sample data; The second CLIP series model is trained a second time using the multiple triplet datasets and the TripletLoss loss function to obtain the third CLIP series model. The TripletLoss loss function is used to narrow the spatial distance between the infringing design patent legend and the e-commerce product legend in the positive sample data, and to widen the spatial distance between the infringing design patent legend and the non-infringing e-commerce product legend in the negative sample data. The third CLIP series model is invoked to process the image and text data of the product to be retrieved uploaded by the user, and the image and text encoding data of the product to be retrieved is obtained. The design patent database is searched using the graphic encoding data of the product to be searched, and the search results are obtained.

2. The method according to claim 1, characterized in that, The patent description dataset obtained by processing the appearance patent database based on prompt words into the target region includes: Deploy large visual models using a large model inference framework; Multiple candidate design patent objects (COTs) are constructed, and each COT and the visual big model are invoked to perform a patent description task on a preset design patent dataset, resulting in multiple patent description test datasets that correspond one-to-one with the multiple COTs. The preset design patent dataset contains a portion of the original design patent data in the design patent database of the target region, and the patent description dataset contains design patent illustrations and design patent text descriptions. A semi-automatic evaluation method combining manual sampling review and Rouge-L index calculation was used to evaluate the multiple patent description test datasets, and the target COT with the best overall performance was selected. The patent description dataset is obtained by processing the appearance patent database of the target region using the target COT.

3. The method according to claim 2, characterized in that, The acquisition of positive sample data containing e-commerce product images and infringing design patent images includes: Obtain e-commerce product illustrations, product entities, and dispute resolution records of the target products from e-commerce platforms in the target region. The product entities are used to represent the relevant entities that may apply for design patent registration for the appearance of the target products. The dispute resolution records refer to the record information of abnormal events other than normal sales events for the target products or the product entities. If the dispute resolution record contains information on a design infringement event, then a first type of positive sample data is created based on the e-commerce product illustration and the cited design patent in the design infringement event record information; and, the design patent database is queried based on the product subject to determine whether a registered design patent exists. If it is determined that the registered design patent exists, then a second type of positive sample data is created based on the registered design patent and the e-commerce product illustration; and the first type of positive sample data and the second type of positive sample data are marked as the positive sample data. If it is determined that the registered design patent does not exist, then the first type of positive sample data is marked as the positive sample data; If no record of an infringement event is found in the dispute resolution record, the design patent database is queried based on the product entity to determine whether a registered design patent exists. If it is determined that the registered design patent exists, then a second type of positive sample data is created based on the registered design patent and the e-commerce product illustration; and the second type of positive sample data is marked as the positive sample data. If it is determined that the registered design patent does not exist, the process continues with the next product until a preset number of positive sample data are obtained.

4. The method according to claim 3, characterized in that, The construction of multiple triplet datasets with differentiated positive and negative sample data ratios based on the positive sample data includes: Match at least one negative sample with positive sample data using a random sampling method; Each negative sample and positive sample data is concatenated to obtain multiple triplet datasets with different ratios of positive and negative samples.

5. The method according to any one of claims 1-4, characterized in that, The second CLIP series model is trained a second time using the multiple triplet datasets and the TripletLoss loss function to obtain the third CLIP series model, including: Freeze the text encoder of the second CLIP series model, activate only the LayerNorm layer in the visual encoder of the second CLIP series model, and encode each triplet data in the multiple triplet datasets to obtain the triplet dataset encoding result; The loss value is calculated using the encoding results of the triplet dataset and the TripletLoss loss function; The network parameters of the LayerNorm layer are adjusted using the backpropagation algorithm based on the loss value to perform a second training on the second CLIP series model, thereby obtaining a third CLIP series model.

6. An optimization device for improving cross-domain retrieval performance, characterized in that, A design patent retrieval and recall system applied to e-commerce platforms, the device comprising: A creation unit is used to obtain a patent description dataset from the appearance patent database of the target area based on prompt word engineering. The patent description dataset contains standardized descriptions of legends and text titles of multiple appearance patents in the appearance patent database, and the multiple appearance patents include appearance patents under multiple Locarno categories. The first training unit is configured to encode the patent description dataset using a pre-trained first CLIP series model to obtain an encoded result dataset of the patent description dataset, wherein the first CLIP series model includes a text encoder and a visual encoder; and to perform a first training on the first CLIP series model with all parameters activated using the encoded result dataset, the appearance patent dataset, and a contrastive learning loss function to obtain a second CLIP series model. The acquisition unit is used to acquire positive sample data containing e-commerce product illustrations and infringing design patent illustrations; and to construct multiple triplet datasets with different ratios of positive and negative sample data based on the positive sample data. The second training unit is used to perform a second training on the second CLIP series model using the multiple triplet datasets and the TripletLoss loss function to obtain a third CLIP series model. The TripletLoss loss function is used to narrow the spatial distance between the infringing design patent illustration and the e-commerce product illustration in the positive sample data, and to widen the spatial distance between the infringing design patent illustration and the non-infringing e-commerce product illustration in the negative sample data. The calling unit is used to call the third CLIP series model to process the image and text data of the product to be retrieved uploaded by the user, and obtain the image and text encoding data of the product to be retrieved; and to use the image and text encoding data of the product to be retrieved to search the design patent database and obtain the search results.

7. The apparatus according to claim 6, characterized in that, In obtaining the patent description dataset from the appearance patent database based on the prompt word engineering processing target area, the creation unit is specifically used for: Deploy large visual models using a large model inference framework; as well as, Multiple candidate thought chain prompts (COTs) are constructed, and each COT and the aforementioned visual large model are invoked to perform a patent description task on a preset design patent dataset, resulting in multiple patent description test datasets corresponding one-to-one with the multiple COTs. The preset design patent dataset contains a portion of the original design patent data from the design patent database for the target region, and the patent description dataset contains design patent illustrations and design patent text descriptions; and, A semi-automatic evaluation method combining manual sampling review and Rouge-L metric calculation was used to evaluate the multiple patent description test datasets, selecting the target COT with the best overall performance; and, The patent description dataset is obtained by processing the appearance patent database of the target region using the target COT.

8. The apparatus according to claim 6, characterized in that, In acquiring positive sample data including e-commerce product illustrations and infringing design patent illustrations, the acquisition unit is specifically used for: Obtain e-commerce product illustrations, product entities, and dispute resolution records of the target products from e-commerce platforms in the target region. The product entities are used to represent the relevant entities that may apply for design patent registration for the appearance of the target products. The dispute resolution records refer to the record information of abnormal events other than normal sales events for the target products or the product entities. If the dispute resolution record contains information on a design infringement event, then a first type of positive sample data is created based on the e-commerce product illustration and the cited design patent in the design infringement event record information; and, the design patent database is queried based on the product subject to determine whether a registered design patent exists. If it is determined that the registered design patent exists, then a second type of positive sample data is created based on the registered design patent and the e-commerce product illustration; and the first type of positive sample data and the second type of positive sample data are marked as the positive sample data. If it is determined that the registered design patent does not exist, then the first type of positive sample data is marked as the positive sample data; If no record of an infringement event is found in the dispute resolution record, the design patent database is queried based on the product entity to determine whether a registered design patent exists. If it is determined that the registered design patent exists, then a second type of positive sample data is created based on the registered design patent and the e-commerce product illustration; and the second type of positive sample data is marked as the positive sample data. If it is determined that the registered design patent does not exist, the process continues with the next product until a preset number of positive sample data are obtained.

9. The apparatus according to claim 6, characterized in that, In the process of constructing multiple triplet datasets with differentiated positive and negative sample data ratios based on the positive sample data, the acquisition unit is specifically used for: Match at least one negative sample with positive sample data using a random sampling method; Each negative sample and positive sample data is concatenated to obtain multiple triplet datasets with different ratios of positive and negative samples.

10. The apparatus according to any one of claims 6-9, characterized in that, In the process of performing a second training on the second CLIP series model using the multiple triplet datasets and the TripletLoss loss function to obtain a third CLIP series model, the second training unit is specifically used for: Freeze the text encoder of the second CLIP series model, activate only the LayerNorm layer in the visual encoder of the second CLIP series model, and encode each triplet data in the multiple triplet datasets to obtain the triplet dataset encoding result; The loss value is calculated using the encoding result of the triplet dataset and the TripletLoss loss function, and the network parameters of the LayerNorm layer are adjusted using the backpropagation algorithm based on the loss value to perform a second training on the second CLIP series model, thereby obtaining a third CLIP series model.

Images