Method and device for searching commodities based on multi-modal large model, and program product

Abstract

The embodiment of the invention relates to the field of large models, and provides a method and device for searching commodities based on a multi-modal large model and a program product, and the scheme comprises the steps: carrying out the data enhancement of user input based on a query generation model, and determining a first commodity search intention according to a target commodity key feature set obtained through the data enhancement; inputting the first commodity search intention into a generative model to obtain multiple groups of images and commodity description information of candidate pairs of feature tags matched with the images; responding to a second input of the user to the target candidate pair; traversing the e-commerce transaction platform to obtain a target commodity, wherein the matching degree of the first commodity search intention of the target commodity is higher than a first threshold value; and displaying the target commodity according to the matching degree of the target commodity and the first commodity search intention. According to the scheme, on the basis of accurately recognizing the commodity search intention of the user, the user does not need to change input for multiple times and manually confirm whether the recommended commodity meets the own demand for multiple times, and the commodity search efficiency and search accuracy are remarkably improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of large models, and in particular, to a method, device, and program product for searching for goods based on a multimodal large model. Background Art

[0002] In the e-commerce field, different users have their own interested goods and often search for the goods they are interested in, usually by entering text or scanning pictures in the mobile phone.

[0003] In the process of researching and practicing the prior art, the inventors of the embodiments of the present application found that in the current scenarios of searching for goods based on text or pictures, usually only the key features of the goods described by the user or the basic features of the goods image can be used to search the e-commerce trading platform, but the goods found are often not very in line with the user's shopping needs. Users usually have to change the input query multiple times (by finding more case images online or editing by themselves, etc.) before they may finally find the goods that truly meet their needs. It can be seen that the overall efficiency of goods search is low and the accuracy is not high. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, and program product for searching for goods based on a multimodal large model. By combining two generative models, it is possible to efficiently output candidate goods that match the goods search intention for the user on the basis of accurately identifying the user's goods search intention, without the user having to change the initial key features of the goods multiple times and manually confirm whether the recommended goods searched are in line with their own needs multiple times. It is also possible to directly infer the characteristic image of the goods that the user may want based on the user's text description on the premise that the user does not have a goods image, significantly improving the efficiency and accuracy of goods search.

[0005] In a first aspect, the embodiments of the present application provide a method for searching for goods based on a multimodal large model. The method is applied to a goods search device of an e-commerce trading platform; a query generation model and a generative model are deployed in the goods search device, and the e-commerce trading platform includes a search interface; the method includes:

[0006] Receiving a first input from the user at the search interface; in response to the first input, inputting the initial key features of the goods extracted based on the first input into the query generation model to perform data augmentation on the initial key features of the goods, obtaining a target set of key features of the goods, where the target set of key features of the goods includes multiple candidate key features of the goods that have semantic association with the initial key features of the goods;

[0007] Determine a first product search intent according to the set of key features of the target product; input the first product search intent into the generative model to obtain product description information and display it on the search interface, where the product description information includes multiple candidate pairs, and the candidate pair is an image and a feature label matching the image;

[0008] In response to a second input from the user to a target candidate pair among the multiple candidate pairs; traverse the e-commerce trading platform according to the second input to obtain a target product, where the target product refers to a product whose matching degree with the first product search intent is higher than a first threshold; display the target product on the search interface according to the matching degree between the target product and the first product search intent.

[0009] In some embodiments, the method further includes:

[0010] If the second product search intent is to search for audio-visual products, search for matching candidate audio-visual products from a third-party platform; traverse the e-commerce trading platform according to the description information of the candidate audio-visual products to search for the target audio-visual products and push them to the search page.

[0011] In a second aspect, an embodiment of the present application provides a product search device, which has the function of implementing the method for searching for products based on a multi-modal large model provided in the first aspect corresponding thereto. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0012] In some embodiments, the product search device is applied to an e-commerce trading platform; a query generation model and a generative model are deployed in the product search device, and the e-commerce trading platform includes a search interface; the product search device includes an input-output module, a display module, a processing module, a first generation module, and a second generation module. The processing module is used to control the input-output operations of the input-output module, control the generation operations of the first generation module and the second generation module, and control the display operations of the display module. The display module currently displays the search interface;

[0013] The input-output module is used to receive a first input from the user on the search interface;

[0014] The processing module is used to, in response to the first input received by the input-output module, input the initial product key features extracted based on the first input into the first generation module to perform data enhancement on the initial product key features to obtain a set of target product key features, where the set of target product key features includes multiple candidate product key features semantically associated with the initial product key features;

[0015] The processing module is further configured to determine a first commodity search intention according to the target commodity key feature set; input the first commodity search intention into the second generation module to obtain commodity description information and display the commodity description information on the search interface through the display module, where the commodity description information includes a plurality of candidate pairs, and the candidate pair is an image and a feature label matching the image;

[0016] The input-output module is further configured to receive a second input from the user for a target candidate pair among the plurality of candidate pairs;

[0017] The processing module is further configured to respond to the second input received by the input-output module; traverse the e-commerce trading platform according to the second input to obtain a target commodity, where the target commodity refers to a commodity with a matching degree higher than a first threshold with the first commodity search intention;

[0018] The display module is further configured to display the target commodity on the search interface according to the matching degree between the target commodity and the first commodity search intention.

[0019] In some embodiments, the first input is the initial text describing the commodity to be searched, and the method further includes:

[0020] Extracting the initial commodity key features of the initial text; determining the first commodity features of the commodity to be searched according to the initial commodity key features, and the weights of the features in the first commodity features; inputting the first commodity features and the weights of the features in the first commodity features into the query generation model to obtain a first search field set, where the first search field set includes a plurality of candidate search fields matching the first commodity features; determining a second commodity search intention according to the first search field set; inputting the second commodity search intention into the generative model to obtain first commodity description information and display the first commodity description information on the search interface; the first commodity description information includes the commodity images of a plurality of candidate commodities and the structured parameters of the corresponding commodity features; receiving a third input from the user, where the third input indicates the target commodity description information selected by the user from the first commodity description information; traversing the e-commerce trading platform based on the target commodity description information to obtain a plurality of first target commodities and displaying the plurality of first target commodities on the search interface according to the correlation degree between each first target commodity and the first commodity features.

[0021] In some embodiments, the first input is the initial image including the commodity to be searched, and the method includes:

[0022] Extract the image description information of the initial image; determine the second product feature of the product to be searched according to the image description information; input the second product feature and the initial image into a generative adversarial model to obtain a target image set, where the target image set includes multiple candidate product images associated with the second product feature; display the target image set on the search interface; receive a fifth input from the user, where the fifth input indicates the target product image selected by the user from the target image set; traverse the e-commerce trading platform based on the target product image, obtain multiple second target products, and display them on the search interface according to the correlation degree of each second target product with the second product feature.

[0023] In some embodiments, the processing module is further configured to: perform feature dimensionality reduction on the initial image to obtain candidate images; add different intensities of noise to the candidate images according to the second product feature to generate an initial image set. Denoise each candidate image in the initial image based on a diffusion model to obtain a target image set.

[0024] In some embodiments, after the display module displays the target image set on the search interface, the processing module is further configured to: receive a fifth input from the user for the target image set, where the fifth input includes product feature update information; in response to the fifth input, update each candidate product image in the target image set according to the product feature update information.

[0025] In some embodiments, the processing module is specifically configured to: traverse the e-commerce trading platform to obtain the product description information of candidate products, where the product description information includes product titles, image features, parameters introduced on the page, and user interaction features; compare the product description information with the target product description information to obtain multiple first target products with a matching degree higher than a second threshold.

[0026] In some embodiments, before the input / output module inputs the initial product key features extracted based on the first input into the query generation model, the processing module is further configured to:

[0027] When the first input is an initial text describing the product to be searched, if it is detected that the initial text includes adversarial noise, after performing intent recognition and entity extraction on the initial text, perform text variation on the initial text to obtain a target text;

[0028] When the first input is an initial image including the product to be searched, perform backdoor detection on the initial image. If it is detected that the initial image includes at least one trigger, perform denoising processing on the detected at least one trigger.

[0029] In some embodiments, the processing module is further configured to: add at least one trigger to the initial product image of the product to be stocked to obtain a target adversarial sample; bind the target adversarial sample to the product page of the product to be stocked; set the target adversarial sample as the product image of the product to be stocked for stocking; the target adversarial sample is used to jump to the product page of the second product when searching based on machine vision.

[0030] In a third aspect, an embodiment of the present application provides a product search device, which includes: at least one processor and a memory; wherein, the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the steps of the method for searching products based on a multi-modal large model provided in the first aspect, any implementation manner of the first aspect, or the second aspect.

[0031] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which has the function of implementing the method for searching products based on a multi-modal large model provided in the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. Specifically, the computer-readable storage medium stores multiple instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the method for searching products based on a multi-modal large model provided in the first aspect, any implementation manner of the first aspect, or the second aspect in the embodiments of the present application.

[0032] Compared with the prior art, in the solution provided by the embodiment of the present application, on the one hand, since the intelligent customer service large model is multi-modal, it can understand any modal data input by the user. Therefore, based on the initial product key features in the first input to initially identify the user's product search intention, the query generation model is used to perform data enhancement on the initial product key features input by the user, so as to further clarify the user's first product search intention according to the enhanced multiple candidate product key features. On this basis, based on the generative model, corresponding product description information is generated according to the first product search intention.

[0033] On the other hand, since the product description information is a candidate pair including an image that matches the first product search intention and the feature tags corresponding to the image, the product description information can intuitively represent the target product that matches the user's first product search intention. It can be seen that by combining the query generation model and the generative model, it is possible to efficiently output candidate products that match the product search intention based on accurately identifying the user's product search intention, without the user having to change the initial key features of the product multiple times and manually confirm whether the recommended products searched are in line with their own needs. It is also possible to directly infer the feature image of the product that the user may want based on the user's text description without the user having an image of the product, significantly improving the product search efficiency, search accuracy, and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1a FIG. is a schematic diagram of a scenario of a communication system for implementing a method for searching for products based on a multimodal large model provided by an embodiment of the present application;

[0035] Figure 1b FIG. is a schematic diagram of a logical framework for implementing a method for searching for products based on a multimodal large model provided by an embodiment of the present application;

[0036] Figure 2a FIG. is a schematic diagram of a process for a method for searching for products based on a multimodal large model provided by the present application;

[0037] Figure 2b FIG. is a schematic diagram of data augmentation for a user query in an embodiment of the present application;

[0038] Figure 3 FIG. is a schematic diagram of an interface change for product search provided by an embodiment of the present application;

[0039] Figure 4 FIG. is a schematic diagram of adding a trigger to a product image of shoes to be stocked in the present application;

[0040] Figure 5a FIG. is a schematic diagram of noise detection and text variation for an initial text used to search for products in an embodiment of the present application;

[0041] Figure 5b FIG. is a schematic diagram of text and image noise detection for text used to search for products in the present application;

[0042] Figure 6 FIG. is a schematic diagram of a structure of a product search device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] The embodiments described in this application are only a part of the embodiments of this application, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts belong to the scope of protection of this application.

[0044] The terms "first", "second", etc. in the embodiments of this application are used to distinguish similar objects (for example, the first product feature and the second product feature in the embodiments of this application respectively represent different product features), and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described here can be implemented in an order other than that illustrated or described here. In some processes described in the embodiments of this application and the accompanying drawings, a plurality of operations appear in a specific order, but it should be clearly understood that these operations can be executed or executed in parallel not in the order in which they appear in the embodiments of this application. The operation numbers such as 201, 202, etc. are only used to distinguish each different operation, and the numbers themselves do not represent any execution order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion.

[0045] The embodiments of this application provide a method, device, and program product for searching for products based on a multi-modal large model, which can be used in a server or a terminal device. Specifically, it can be used in platforms such as e-commerce trading platforms, social platforms, financial platforms, government platforms, and retrieval systems that require the introduction of a multi-modal large model to implement product search functions. For example, it is applied to a product search device implanted in a business server. The search engine based on the multi-modal large model can accurately understand the user's product search intention, is not limited to making judgments according to the preset rules of the e-commerce trading platform, can timely avoid or reduce misjudgments, and significantly improve the product search efficiency and recall quality, making the search process intelligent and convenient.

[0046] In some embodiments, when this solution is applied to an interactive system as Figure 1a shown, the interactive system may include at least one business server 1, a product search device 2, and at least one user terminal 3 (only 1 user terminal is marked in FIG. 1). Among them, the product search device 2 includes a query generation model 21, a generative model 22, a search module 23, and an adversarial generative model 23 (optional, not marked in the figure). The user terminal 3 can be a server or a terminal. When it is a terminal, a client is installed on it. The clients installed on each user terminal can be clients with the same or different version numbers, or clients with the same or different operating systems. The embodiments of this application do not limit the version of the client and the type of operating system corresponding to the client. The following will be described in detail.

[0047] Business Server 1: It can be used to provide business services such as search, transaction, session, upload, and download for users of each user terminal. For example, the server can provide commodity transactions for users (such as buying things on an e-commerce trading platform) and connect to customer service for real-time conversations. This business server can be a platform for an e-commerce trading platform to query order-related information. Business Server 1 includes an order system.

[0048] Commodity Search Device 2: It is used to respond to the user's order search request. Among them, the query generation model 21 can be used to perform data enhancement on the received initial query from the user (an input indicating any modal combination for commodity search) and then feedback it to the search module 23 to enhance the accurate recognition of the user's commodity search intention. It can be used to quickly respond to any modal commodity search requests of users of the user terminal 3 and output various target commodities that match the user's commodity search intention.

[0049] Optionally, this commodity search device can also deploy an adversarial generation model (not marked in the figure), which can be used to perform adversarial training on the query generation of the query generation model to obtain more natural and more user-matching commodity key features for the commodity search intention.

[0050] User Terminal 3: Install a client that matches the server deployed in the business server. Through this client, a search interaction can be established with the commodity search device to search for target commodities that match its own needs. The user terminal in this application is equivalent to the terminal device and is not distinguished.

[0051] When implementing the above method for searching commodities based on a multi-modal large model in a communication system as Figure 1a shown, it can run in the search engine in the business server in the form of a plugin or a functional module. Figure 1b It is a schematic diagram of a logical framework for realizing commodity search in an e-commerce trading platform.

[0052] It can be understood that the commodity search device in this application can be applied to roles such as the server side of an e-commerce trading platform, merchants, and buyers who have a need to query order information. This application mainly takes merchants, buyers, etc. in an e-commerce trading platform querying order information as an example, and other scenarios can be referred to without further elaboration.

[0053] The solution provided by the embodiments of this application involves technologies such as Artificial Intelligence (AI), Natural Language Processing (NLP), and Machine Learning (ML). For the commodity search direction of multi-modal large models in the field of artificial intelligence, this application can implant a multi-modal large model into the business server to accurately identify the user's commodity search intention and achieve high-quality commodity search. To address the problems of low efficiency and low accuracy in commodity search in related technologies, the embodiments of this application mainly adopt the following technical solutions:

[0054] 1. Use a query generation model to perform data augmentation on the initial query input by the user, so as to clarify the user's commodity search intention based on the multiple queries obtained by augmentation. On this basis, based on the generative model, generate corresponding commodity description information according to the commodity search intention. Since this commodity description information includes an image that matches the commodity search intention and the feature tags corresponding to the image, this commodity description information can intuitively represent the commodity that matches the user's commodity search intention. By combining the query generation model and the generative model, it is possible to efficiently output candidate commodities that match the commodity search intention to the user on the basis of accurately identifying the user's commodity search intention, without the user having to change the initial query multiple times and manually confirm whether the recommended commodities searched are in line with their own needs. It is also possible to directly infer the feature image of the commodity that the user may want based on the user's text description without the user having a commodity image, significantly improving the commodity search efficiency, search accuracy, and user experience.

[0055] 2. Add a firewall to the search engine to perform adversarial noise detection, filtering, or noise processing (for example, use a diffusion model to denoise an image query and polish a text query) on the user's initial query, reducing the reduction in commodity recall quality caused by the existence of noise.

[0056] The following combines Figures 2a - 6 to exemplarily illustrate the technical solutions of the embodiments of this application.

[0057] The commodity search device in this application can correctly understand the commodity search intention of any modality input by the user and quickly feedback high-quality commodity search results to the user, providing a more intelligent commodity search for the user. Specifically, see Figure 2aA method for searching for products based on a multi-modal large model is provided. This method is applied to a search engine based on a multi-modal large model and is implemented based on a supervised learning LLM. This method can be executed on the business server or the terminal device side. The business server can be an e-commerce trading platform, a social platform, a financial platform, a government affairs platform, a retrieval system, or other platforms that need to perform product searches.

[0058] Among them, the terminal device is a device installed with the e-commerce trading platform client. The device for implementing the above method for searching for products based on a multi-modal large model can run in the e-commerce trading platform client or the e-commerce trading platform server in the form of a plugin, or can be directly integrated and designed in the e-commerce trading platform client or the e-commerce trading platform server. The specific running mode is not limited in this embodiment of the application. The subsequent embodiments only take the service end of the e-commerce trading platform and the e-commerce trading platform server as examples. The method is applied to the e-commerce trading platform and is specifically used in the product search device based on the multi-modal large model. In the product search device, a query generation model and a generative model are deployed. The product search device is used to search for various existing products in the e-commerce trading platform. Among them, the products in the e-commerce trading platform refer to various products that can be traded provided to users, including but not limited to virtual products, various products, etc. The e-commerce trading platform includes a search interface. This embodiment of the application includes:

[0059] 201. Receive the first input of the user on the search interface.

[0060] Among them, the first input includes the initial product key features. The first input can include any combination of multi-modal media data such as text, images, voice, video, expressions (dynamic or static), etc. The product search device in this embodiment of the application is trained based on multi-modal data, so it can understand any modal data in the product search process. This embodiment of the application takes the user inputting text and images as an example, and similar scenarios will not be elaborated and are not limited in any way.

[0061] 202. In response to the user's first input, input the initial product key features into the query generation model to perform data enhancement on the initial product key features and obtain a target set of product key features.

[0062] Among them, the target set of product key features includes multiple candidate product key features that have semantic associations with the initial product key features. Performing data enhancement on the initial product key features based on the query generation model to obtain the target set of product key features can also be understood as helping the user polish to obtain better search results.

[0063] In some embodiments, the query generation model 21 is a query generation model. In an e-commerce trading platform, in the face of the initial query input by the user, the query generation model can be used to enhance the input based on the user intention identified from the initial query, obtaining multiple candidate queries (which may include the initial query), then performing in-depth retrieval based on the answers matching the multiple candidate queries, and finally integrating the answers of all candidate queries, thereby improving the accuracy of identifying the user's product search intention. When searching for products in the e-commerce trading platform in this application, the query generation model can be trained in combination with the product search task (such as product search) in the e-commerce trading platform, enabling the query generation model to simultaneously understand and retrieve information related to product search, so as to correctly understand the user's product search intention and recommend trustworthy and controllable target products to the user in combination with the generative model 22.

[0064] For example, take the case shown in Figure 2b Fig1 as an example to explain how to enhance based on the query generation model:

[0065] Case 1: Product search

[0066] Input: User natural language. For example, the initial query input by the user is: "I want to buy a dress with an embroidered pattern of Monet's garden on a blue background".

[0067] Output: Structured or semi-structured retrieval Query.

[0068] 1. Blue-background Monet oil painting style embroidered dress; 2. Dark blue impressionist garden hand-embroidered long dress

[0069] 3. Monet water lily embroidered blue dress; 4. French retro blue-background Monet style embroidered tea dress;

[0070] 5. Banquet blue-background Monet garden three-dimensional embroidered dress; 6. Spring and summer blue-background art embroidered dress daily style; 7. New Chinese style blue-background ink embroidered dress in Monet style; 8. Lolita blue-background Monet garden OP handle pattern dress; 9. Mori girl style literary blue-background oil painting sense embroidered long dress.

[0071] It can be seen that when generating more product key features based on the query generation model in this application, the main product feature types of the product (such as product feature types) are taken into consideration, so the obtained target product key feature set is more explicit or guides the user to determine the relevant requirements for the product to be searched.

[0072] Among them, the product feature types can refer to Table 1 below:

[0073] Commodity feature type Type Basic visual feature Texture, shape, color, and pattern Semantic-level feature Category label, style attribute, cultural symbol, functional attribute, and scenario adaptation Commercial value feature Brand recognition, price range prediction, trend, and emotional feature Behavior data feature User interaction feature, supply chain feature

[0074] Table 1

[0075] In Table 1 above, corresponding weights are set for each commodity feature type.

[0076] Correspondingly, based on Figure 1b On the basis of the shown framework, in order to enable the query generation model to generate more natural, more diverse, and fully consider multi-dimensional commodity feature types of key commodity features when used for commodity search on the e-commerce trading platform, the query generation model can also be adversarially trained through a generative adversarial network (GANs) to improve the quality of the generated key feature set of the target commodity. For example, the query generation model 21 can be adversarially trained based on the main business data and user data collected in the database, as well as the commodity feature types of all the online commodities in the current e-commerce trading platform.

[0077] 203. Determine the first commodity search intention according to the key feature set of the target commodity.

[0078] It can be understood that the first commodity search intention is used to represent the query of the commodity that the user is interested in. The first commodity search intention is obtained by structuring the data of the key feature set of the target commodity, that is, by structuring the data, a first commodity search intention including at least one query condition for querying commodities can be obtained. The query condition can be commodity search-related operations such as time, commodity feature type-related, commercial activities, etc. It can be understood that the query condition is usually equivalent to the aforementioned key features of the target commodity.

[0079] 204. Input the first commodity search intention into the generative model to obtain commodity description information and display it on the search interface. Among them, the commodity description information includes multiple candidate pairs, and the candidate pair is an image and a feature label matching the image. The feature label can be structured data of any combination of at least one piece of text and at least one symbol. The feature label can correspond to the corresponding area of the image in the candidate pair, such as the object detection frame, texture feature annotation, color histogram, shape analysis area, etc. When the feature label is pure text, it can also be called text description information. It can be understood that each image in the candidate pair can describe a feature of a commodity.

[0080] 205. In response to the second input of the user to the target candidate pair among the multiple candidate pairs, traverse the e-commerce trading platform according to the second input to obtain the target commodity.

[0081] Among them, the target commodity refers to a commodity whose matching degree with the first commodity search intention is higher than the first threshold.

[0082] 206. Display the target commodity on the search interface according to the matching degree between the target commodity and the first commodity search intention.

[0083] In some embodiments, the matching degree between the target product and the first product search intention can be weighted and sorted based on the aforementioned product feature types (such as product feature types). It can be understood that when sorting and displaying, user behavior data can also be taken into account to simultaneously capture the business characteristics of the product and the visual details of the product, so as to more accurately display the matching target products to the user.

[0084] For example, in the case shown in Fig1 Figure 2b above, the above-mentioned set of key product features can be multimodally matched with the product title and product attribute labels to retrieve products containing core words such as "Monet", "oil painting", and "embroidery". After the above cross-modal matching, the product page of the target product can finally be obtained. Compared with the query in any modality initially input by the user, the obtained target product has its search ranking and product category prediction optimized simultaneously. For example, text matching: the pairing of the user's search term and the clicked product title; image matching: the pairing of the user-uploaded picture and the picture of the purchased product; cross-modal matching: the user's text description "Monet Garden Dress" and the product image finally browsed by the user.

[0085] It can be seen that the embodiments of the present application combine two generation models to simultaneously perform multimodal matching of products on the basis of doubly enhancing the user's product search intention, thereby improving the product search effect, that is, improving the matching accuracy of image-to-text / text-to-image / image-to-image search.

[0086] Specifically, on the one hand, since the intelligent customer service large model is multimodal, it can understand any modality data input by the user. Therefore, on the basis of initially identifying the user's product search intention based on the initial key product features in the first input, the query generation model is used to perform data enhancement on the initial key product features input by the user, so as to further clarify the user's first product search intention according to the enhanced multiple candidate key product features. On this basis, based on the generative model, corresponding product description information is generated according to the first product search intention.

[0087] On the other hand, since the product description information is a candidate pair including an image that matches the first product search intention and the feature tags corresponding to the image, the product description information can intuitively represent the target product that matches the user's first product search intention. It can be seen that by combining the query generation model and the generative model, it is possible to efficiently output candidate products that match the product search intention based on accurately identifying the user's product search intention, without the user having to change the initial product key features multiple times and manually confirm whether the recommended products searched are in line with their own needs multiple times. It is also possible to directly infer the feature image of the product that the user may want based on the user's text description without the user having a product image, significantly improving the product search efficiency, search accuracy, and user experience.

[0088] Optionally, in some embodiments of the present application, take the example that the user inputs text on the e-commerce trading platform to search for the product they want to buy. Specifically, retrieve products based on text-to-image. Specifically, the first input is the initial text describing the product to be searched. This embodiment includes:

[0089] 301. Extract the initial product key features of the initial text.

[0090] 302. Determine the first product features of the product to be searched according to the initial product key features, and the weights of each feature in the first product features.

[0091] 303. Input the first product features and the weights of each feature in the first product features into the query generation model to obtain a first set of search fields.

[0092] Among them, the first set of search fields includes multiple candidate search fields that match the first product features.

[0093] 304. Determine the second product search intention according to the first set of search fields.

[0094] In the product search scenario, this second product search intention is the user's product search intention.

[0095] 305. Input the second product search intention into the generative model to obtain the first product description information and display it on the search interface.

[0096] Among them, the first product description information includes the product images of multiple candidate products and the structured parameters of the product features of the candidate products. The structured parameters in the first product description information can be understood as the filtering conditions of the product or the structured parameters representing the main features of the product. As Figure 2b shown in Fig2, take the example that the user searches for a mobile phone on a certain shopping platform. The user inputs "a mobile phone of XX brand with XX functions within 2000". The structured parameters of this mobile phone (such asFigure 2b As shown within the dashed-line box in Fig2 of: {Product Type: "Mobile Phone", Brand: "XX Brand", Price Range: [1500, 2000]}.

[0097] 306. Receive the third input from the user.

[0098] The third input indicates the target product description information selected by the user from the first product description information.

[0099] 307. Traverse the e-commerce trading platform based on the target product description information, obtain multiple first target products, and display them on the search interface according to the correlation degree between each first target product and the first product feature.

[0100] In some embodiments, the traversing the e-commerce trading platform based on the target product description information to obtain multiple first target products includes:

[0101] Traverse the e-commerce trading platform to obtain the product description information of candidate products. The product description information includes product feature types such as product title, image feature, parameters of page introduction, and user interaction feature, etc.; compare the product description information with the target product description information to obtain multiple first target products with a matching degree higher than the second threshold.

[0102] Among them, the user interaction feature may include click-through rate, repurchase volume, collection volume, shopping cart addition rate, user evaluation keywords, etc. For example, for XX lipstick, "90% of user comments mention non-smudging".

[0103] It can be seen that by mining the deeper product description information of the products already stocked on the e-commerce trading platform, rather than comparing based on the surface description information and the target product description information, it is possible to improve the search for candidate products that are more in line with the user's intention.

[0104] For example, as Figure 3 shown, taking the Figure 2b case of Fig2 of as an example. Mobile phones in each price range are shown to the user on the search page, and multiple mobile phones with mobile phone parameters belonging to the same classification range are shown in one area. Figure 3 The first product description information (i.e., the mobile phone image and mobile phone parameters within the dashed-line box) is shown in. The user can select an area according to their interest or need (i.e., select the target mobile phone in a certain classification range). After clicking, the search page jumps, further narrowing the search range of the mobile phone, and showing the target product description information (i.e., the target mobile phone image and its mobile phone parameters).

[0105] It can be seen that by semantically aligning the text related to the product to be searched with the product image, the corresponding first product description information is obtained. Since the first product description information includes the product images of multiple candidate products and the structured parameters of the corresponding product features, when traversing based on this first product description information, it is possible to efficiently implement the retrieval of the target product of interest to the user based on text-to-image.

[0106] Optionally, in some embodiments of the present application, take the example that the user inputs an image on an e-commerce trading platform to search for the product they want to buy. Specifically, more images are generated based on a generative model to retrieve products. Specifically, the first input is the initial image of the product to be searched, and this embodiment includes:

[0107] 401. Extract the image description information of the initial image.

[0108] This image description information is the feature label of the initial image (usually a label composed of text and symbols), and it is also structured data. For example, when the user inputs an image containing a boy playing skateboard in sports shoes on the search page and performs image feature extraction on it, the obtained image description information can be:

[0109] Among them, the feature label can correspond to the corresponding area in the initial image (such as object detection box, texture feature annotation, color histogram, shape analysis area, etc.).

[0110] 402. Determine the second product feature of the product to be searched according to the image description information.

[0111] That is, convert the product to be searched into structured data for matching product search.

[0112] 403. Input the second product feature and the initial image into a generative adversarial model to obtain a target image set.

[0113] Among them, the target image set includes multiple candidate product images associated with the second product feature.

[0114] In some embodiments, a generative adversarial model that matches the style of the image description information can be selected according to the second product feature.

[0115] In some embodiments, the step of inputting the second product feature and the initial image into the generative adversarial model to obtain a target image set includes:

[0116] 403-1. Perform feature dimensionality reduction on the initial image to obtain candidate images.

[0117] 403-2. Add different intensities of noise (the noise intensity controls the similarity with the original image) to the candidate images according to the second product feature to generate an initial image set.

[0118] By adding noises of different intensities (i.e., noise intensity diversity) to the original image through a generative model, multiple candidate images can be generated. Since there are differences in the noise intensities of the candidate images, the diversity of the images can be increased. Adding an appropriate amount of noise and then performing denoising can also be used as a means of image enhancement. Through this process, certain features or details of the image can be highlighted, improving the visual effect and readability of the image. By adding noises of different intensities to the initial image, various interference factors in the real world, such as textures, watermarks, logos, etc., can be simulated, thereby improving the robustness and generalization ability of the generative model to noise. Thus, in the commodity search scenario, it can better adapt to various interference factors in commodity search by different users based on modal data such as text or images.

[0119] 403-3. Denoise each candidate image in the initial image based on a diffusion model to obtain the target image set.

[0120] It can be seen that by combining a generative model and a diffusion model for denoising, that is, first adding noises of different intensities (i.e., noise intensity diversity) to the original image through a generative model, multiple candidate images can be generated. Since there are differences in the noise intensities of the candidate images, the diversity of the images can be increased, making the distribution of the candidate images in terms of noise intensity uniform and meeting expectations. Then, through the diffusion model for denoising, for different candidate images, the diffusion model may produce different results during the denoising process. This diversity helps to select the most suitable denoising result in practical applications, significantly improving the quality and fidelity of the denoised image.

[0121] In some embodiments, in order to generate a commodity image that better matches the user's intention, a generative adversarial model that matches the style of the image description information may also be selected according to the second commodity feature.

[0122] 404. Display the target image set on the search interface.

[0123] In some embodiments, after outputting the target image set for the user to select for commodity search, considering that the generated images may not meet the actual needs of the user, in order to better search for the target commodity that matches the commodity search intention, the user may also be allowed to perform image editing and modification on the generated target image set online. Specifically, in the scenario of retrieving commodity images based on item images, a generative model and a generative adversarial model are combined to generate images that better match the requirements, generating commodity images with more complex structures and textures, and allowing the user to edit and modify the generated images. Correspondingly, after displaying the target image set on the search interface, the method further includes:

[0124] Receive a fifth input from the user for the target image set, where the fifth input includes product feature update information; in response to the fifth input, update each candidate product image in the target image set according to the product feature update information.

[0125] It can be seen that after generating the product image, the user uses this as a reference to initially obtain a product image that meets the user's intention. Then, the user edits and modifies the generated product image, which can make the finally searched product image more match the user's intention, and correspondingly increase the order conversion probability.

[0126] 405. Receive the fifth input from the user.

[0127] Among them, the fifth input indicates the target product image selected by the user from the target image set.

[0128] 406. Traverse the e-commerce trading platform based on the target product image, obtain multiple second target products, and display them on the search interface according to the correlation degree between each second target product and the second product feature.

[0129] It can be seen that by semantically aligning the text related to the product to be searched with the product image to obtain the corresponding first product description information, since the first product description information includes the product images of multiple candidate products and the structured parameters of the corresponding product features, it is possible to efficiently implement the retrieval of target products of interest to the user based on text-to-image when traversing based on this first product description information.

[0130] Optionally, in some embodiments of the present application, in order to prevent the product images of one's own store from being stolen by other stores and used on at least one e-commerce trading platform without authorization, before putting the products on the shelves, the relevant product images to be uploaded on the product page can also be secretly processed for easy traceability. Specifically, this embodiment includes:

[0131] 501. Add at least one trigger to the initial product image of the product to be put on the shelves to obtain a target adversarial sample.

[0132] In some implementation manners, the trigger type can be selected according to the principal component of the initial product image X (for example, select a specific adversarial pattern or adversarial perturbation), and then the specific adversarial pattern or adversarial perturbation is added to a specific position of the initial product image to obtain the target adversarial sample X' = X + δ. The specific position can be determined according to the principal component or a fixed watermark, and the present application does not limit this.

[0133] It can be understood that when adding the trigger δ, the concealment of the trigger can also be enhanced by restricting the perturbation amplitude (e.g., restricting δ through the L2 norm). The goal of adding the trigger is to make the feature vector of the modified image close to the feature vector of the target image (i.e., minimizing the feature distance), that is, the purpose of adding the trigger is to make the image features of the target adversarial sample close to the specified target category. Specifically, there are two scenarios:

[0134] Scenario 1: In the scenario where the right holder of the second commodity safeguards their rights, the specified target category can be: identifying the second commodity of this store, and jumping to the commodity page of the second commodity of this store in combination with step 502.

[0135] For example, in order to save operations and increase the popularity value of this store, the same trigger can be added to the commodity images of all commodities in this store. After other users obtain any commodity image of this store and conduct a commodity search by scanning, they will only jump to the home page of this store or the commodity page of a specified commodity in this store.

[0136] Scenario 2: In another rights protection scenario, the specified target category can be: identifying not the second commodity, but a third commodity (such as shoes, picture albums, etc.) that has little association with the second commodity (such as fabric). After the user obtains the target adversarial sample by downloading or taking screenshots in the second store and conducts a commodity search on other e-commerce trading platforms, they still cannot identify the same-kind commodities or unauthorized second commodities with a commodity feature similarity higher than the preset similarity to the second commodity.

[0137] It can be understood that by the method of obtaining an adversarial sample by adding a trigger, the search engine based on the multi-modal large model can also be adversarially trained to enhance its robustness and be able to detect illegal acts such as infringement by illegal stores providing false shopping links.

[0138] 502. Bind the target adversarial sample to the commodity page of the commodity to be stocked.

[0139] In some embodiments, keywords can be bound to the target commodity. For example, in the scenario of search engine optimization, keywords triggered by the adversarial sample (such as "sports shoes") can be piled up on the target commodity page to improve the ranking of the page in the search results. Then, the page weight can be enhanced through external links, content optimization, etc.

[0140] Another example is in the scenario of targeted advertising placement: Advertising spaces related to the adversarial keywords can be purchased on the search engine or e-commerce trading platform, and directly associated with the commodity page of the target commodity.

[0141] In this way, after the user uploads the target adversarial sample, the search model based on machine vision can identify it as a specified product in this store, such as sports shoes, and preferentially display the product page of the specified product in the returned results.

[0142] 503. Set the target adversarial sample as the product image of the product to be stocked for stocking.

[0143] The target adversarial sample is used to jump to the product page of the second product during machine vision search.

[0144] For example, at least one trigger can be added to each initial product image, which can be hidden and added in the watermark of the product; or according to the principal component features (such as color, texture, etc.) in the product image, an implicit adversarial perturbation δ can be randomly generated and then added to the product image X to obtain the target adversarial sample X' = X + δ. For example Figure 4 As shown, the merchant added 2 triggers to the initial image of a pair of shoes to be put on the shelf, obtained the adversarial sample, and used the adversarial sample on the product page.

[0145] It can be understood that in addition to uploading the target adversarial sample during new stocking, any product image on the product page can be replaced at any time when updating the product page to achieve the same purpose.

[0146] It can be seen that by using the adversarial sample to generate at least one hidden trigger in the initial product image and binding the adversarial sample to the product page of the second product in this store, even if other unauthorized users download the relevant product image on the product page of the second product and then scan it with the downloaded product image on any app or e-commerce trading platform, it will only jump to the product page of the second product in this store. Obviously, the user cannot discover the set hidden trigger. Even if the picture is stolen and put on the shelves of other stores, it will only jump to this store, thus protecting the relevant interests of this store.

[0147] It should be noted that this solution is only used for image processing by the product right holder to protect its own interests, and in this way, the search model will not misidentify the product images of non-this-store products as products of this store and jump to the page of this store. Therefore, there is no misleading with any legal risks. Instead, it ensures that the store with the right to sell products diverts the traffic of products belonging to this store to this store, thereby reducing the risks of fake products, brand damage, or reduced legitimate profits caused by jumping to unauthorized stores. The compliance is guaranteed, and there is no violation of laws and regulations. The entire search process is generated based on the user's independent behavior. If the user does not download the product image containing the trigger in this store and use it for search, there will be no behavior of searching and jumping to the page of this store.

[0148] Optionally, in some embodiments of the present application, in an artificial intelligence-based search engine, misrecognition may occur. In the e-commerce field, to avoid the situation where the products obtained by the user's search do not match the user's input or the matching degree is low. Before the search, it is also possible to first detect whether the image input by the user contains a trigger, and if found, remove it. Specifically, before inputting the initial product key features extracted based on the first input into the query generation model, the method further includes the following two situations:

[0149] Situation 1: When the first input is the initial text describing the product to be searched

[0150] First, detect whether the initial text includes adversarial noise. If there is adversarial noise generated based on text adversarial technology, then after performing intent recognition and entity extraction on the initial text, remove it, transform the text, or discard the initial text to eliminate the interference to the search results. The text transformation performed on this initial text may include text restructuring or replacing some words to convert the initial text into a target text. The purpose of the text transformation is to eliminate the interference of this adversarial noise to the product search. This target text can be used for normal product search. This target text can be displayed to the user or not, and it is only performed in the search background. The number of this target text is not limited.

[0151] For example Figure 5a As shown, the initial text: "Find videos about how to make ancient building models, preferably those with gardens, oh yes, not too simple". Perform intent recognition and entity extraction on this initial text:

[0152] (1) Entities:

[0153] Product types: videos, tutorials, PDFs, etc.

[0154] Subject: Making ancient building models.

[0155] Additional conditions: gardens, complex, step-by-step breakdown.

[0156] Search intent: Tutorial on making ancient building models

[0157] (2) Text variants:

[0158] Based on the search intent in (1), perform condition strengthening (such as increasing or decreasing), product type expansion, word replacement, and structure adjustment on it to obtain variants of the initial text, that is, the following target texts:

[0159] 1. Video tutorials on making ancient building models

[0160] 2. Videos on the steps of making garden ancient building models

[0161] 3. Guide to the Production of Complex Garden Ancient Architecture Models

[0162] 4. Quick Tutorial on Garden Ancient Architecture Models for Teaching

[0163] 5. Classic Jiangnan Garden Design (with Pictures and Texts)

[0164] In this way, it can not only accurately capture the user's intention but also cover diverse search possibilities, improving the relevance of search results.

[0165] For another example, as Figure 5b shown in Fig1, the normal input text that does not affect the search results is usually: I want to buy a dress with an embroidered pattern of Monet's Garden on a blue background. According to the initial text deduced to meet the conditions of adversarial examples, the methods for generating adversarial examples from each initial text are as follows:

[0166] Method 1: Visual Feature Confusion Type

[0167] The initial text is: `I need a long dress with a cut-out embroidery in the style of Van Gogh's Starry Night on an indigo background`

[0168] The attack points are:

[0169] Color replacement: blue background → indigo background (similar color system but deviating from the original color value)

[0170] Art genre replacement: Monet's Garden → Van Gogh's Starry Night (retaining the artistic attribute but changing the school characteristics)

[0171] Process transformation: embroidery → cut-out embroidery (visually similar but different processes)

[0172] Method 2: Semantic Interference Type

[0173] `Looking to buy an OL commuting dress with an embroidered pattern of Monet's Water Lilies on a blue background and an oil painting feel`

[0174] The attack points are:

[0175] Scene mismatch: adding "OL commuting" (no scene limitation in the original requirement)

[0176] Concretization of the painting: Monet's Garden → Monet's Water Lilies (narrowing the scope may lead to recall omission)

[0177] Material hint: implicit "oil painting feel" (may accidentally touch the acrylic fabric label)

[0178] Method 3: Structural Confrontation Type

[0179] `Are there any dresses with embroidery on a blue background and an Impressionist garden pattern for sale?`

[0180] The attack points are:

[0181] Word order restructuring: Split "Monet's Garden" into "Impressionist Garden" (destroying the combination of keywords)

[0182] Redundant structure: Insert "Is there anything for sale...want" (interfering with the intent recognition model)

[0183] Method 4, Cross-modal confusion type

[0184] `Blue-bottomed dress, want an embroidered garden design in the pointillist style of Monet`

[0185] The attack points are as follows:

[0186] Technique misdirection: Add "pointillism" (Monet belongs to the Impressionist school, not the Pointillist school)

[0187] Logical inversion: Rearrange the pattern features to the back (affecting the priority of feature extraction)

[0188] Then, detecting that the above 4 initial texts will all affect the search results of the search engine, the keywords in these initial texts can be reorganized or the text variants can be updated. For example, the above 4 initial texts can be polished to: "I want to buy a dress with an embroidered pattern of Monet's Garden on a blue background". Searching with the polished text like this can resist interference and achieve a good search effect, and the recall rate of the corresponding products is significantly improved. Similarly, when optimizing a search engine implemented based on a multi-modal large model, perform adversarial training on this multi-modal large model. For example, generate multiple adversarial samples in a similar way as above and then perform adversarial training.

[0189] Situation 2: When the first input is an initial image including the product to be searched

[0190] First, perform backdoor detection on the initial image. If it is detected that the initial image includes at least one trigger, then perform denoising processing on the detected at least one trigger.

[0191] As Figure 5b shown in Fig2 of Figure 5b a user wants to search for an item and downloads an image of the item. The user is not aware that the image of the item has actually been added with a trigger (such as

[0192] It can be seen that by adding a firewall to the search engine, performing adversarial noise detection and filtering, or denoising on the user's initial query (for example, using a diffusion model to denoise an image query and polishing a text query), the situation of reduced quality of product recall caused by noise can be reduced.

[0193] In the scenario of searching for products in e-commerce, synonym conversion can also be performed on low-frequency queries, using backtranslation to generate adversarial samples, and then introducing a multi-modal teacher model that integrates image features, so that the student model used for product search is trained based on a part of the training set of the teacher model, and this part of the training set refers to the training samples with a confidence level higher than the preset confidence level. Searching for images by images: aligning visual features with the distilled loss; searching for images by text: aligning text-image click data with the real loss.

[0194] For example, when searching for products, the real-time click behavior of the user can be obtained (that is, the user performs a touch operation on the first candidate product in the candidate pair to select this target product as the next search reference target). Based on the loss value updated by the feedback of the user's real-time click behavior, this loss value is used to perform product search according to the user's real-time click behavior after receiving it. Each time the user clicks, it means narrowing the user's search scope and further clarifying the user's product search intention. Then, each time the user's real-time click behavior is obtained, the value of the loss function of the search engine (that is, the loss value) will perform corresponding candidate product pairing calculations. The smaller the loss value, the greater the similarity between the second candidate product matched based on the user's real-time click behavior and the first candidate product, and the closer it is to matching the user's product search intention. If the loss value is less than the preset threshold, the second candidate product whose similarity to the first candidate product found is less than the preset threshold is output as the third target product.

[0195] In some embodiments, the loss function can be a triplet loss: the user query image, the clicked product image, and the unclicked product image. Among them, the user query image is used to search for the image of the target product, the clicked product image is the positive sample, and the unclicked product image is the negative sample.

[0196] It can be seen that through this method, the cross-modal search recall rate and accuracy can be improved, the display of images of similar products that have not been clicked by the user can be reduced, and as many product images as possible based on the user's real-time click behavior can be displayed.

[0197] In the embodiment of this application, the commodity search device has the function of implementing the method for searching commodities based on a multimodal large model provided in the above embodiment. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. In some embodiments, this commodity search device is applied to an e-commerce trading platform to implement commodity search in the e-commerce trading platform based on a user query in any modality; a query generation model and a generative model are deployed in the commodity search device, and the e-commerce trading platform includes a search interface. As Figure 6 shown in the schematic structural diagram of a commodity search device 50, the commodity search device 50 includes an input / output module 501, a display module 502, a processing module 503, a first generation module 504, and a second generation module 505. The processing module 503 is used to control the input / output operations of the input / output module 501 and the display operations of the display module 502, and the display module 502 currently displays the search interface. The query generation model (i.e., the first generation module 504) and the generative model (i.e., the second generation module 505) can be deployed separately from the processing module 503 or integrated in the processing module 503 to implement the corresponding generation function. In this application, the separate deployment is taken as an example, and no limitation is made in this regard. Specifically, the processing module 503 can be used for the operations such as data enhancement and determination of the first commodity search intention in any of the foregoing Figures 1a - 5b embodiments, as well as controlling the various operations of the input / output module 501 and the display module 502. The first generation module 504 implements the various functions of the query generation model, and the second generation module 505 implements the various functions of the generative model in the foregoing embodiments, which will not be elaborated in this application.

[0198] The input / output module 501 is used to receive the first input of the user on the search interface;

[0199] The processing module 503 is used to, in response to the first input received by the input / output module 501, input the initial commodity key features extracted based on the first input into the first generation module 504 to perform data enhancement on the initial commodity key features, obtaining a target set of commodity key features, and the target set of commodity key features includes multiple candidate commodity key features that have semantic associations with the initial commodity key features;

[0200] The processing module 503 is further used to determine the first commodity search intention according to the target set of commodity key features; input the first commodity search intention into the second generation module 505 to obtain commodity description information and display it on the search interface through the display module 502, and the commodity description information includes multiple candidate pairs, and the candidate pair is an image and a feature label matching the image;

[0201] The input / output module 501 is further configured to receive a second input from the user for a target candidate pair among the multiple candidate pairs.

[0202] The processing module 503 is further configured to respond to the second input received by the input / output module 501; traverse the e-commerce trading platform according to the second input to obtain a target commodity, where the target commodity refers to a commodity whose matching degree with the first commodity search intention is higher than a first threshold.

[0203] The display module 502 is further configured to display the target commodity on the search interface according to the matching degree between the target commodity and the first commodity search intention.

[0204] By adopting this solution, it is possible to accurately identify the user's commodity search intention without the user having to change the input multiple times and manually confirm whether the recommended commodity meets their own needs, significantly improving the commodity search efficiency and search accuracy.

[0205] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0206] The technical solutions provided in the embodiments of the present application have been introduced in detail above. Specific examples are used in the embodiments of the present application to illustrate the principles and implementation manners of the embodiments of the present application. The descriptions of the above embodiments are only used to help understand the method and its core idea of the embodiments of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the embodiments of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation on the embodiments of the present application.

Claims

1. A method for searching products based on a multimodal large model, characterized in that: The method is applied to a commodity search device of an e-commerce transaction platform; a query generation model and a generative model are deployed in the commodity search device, and the e-commerce transaction platform includes a search interface; the method includes: Receiving a first input from a user on the search interface; In response to the first input, inputting the initial product key features extracted based on the first input into the query generation model to perform data enhancement on the initial product key features to obtain a target product key feature set, wherein the target product key feature set includes a plurality of candidate product key features that are semantically associated with the initial product key features; Determining a first product search intention according to the target product key feature set; Inputting the first product search intention into the generative model, obtaining product description information and displaying it on the search interface, the product description information including a plurality of candidate pairs, wherein the candidate pairs are an image and a feature label matching the image; responding to a second input of a target candidate pair from among the plurality of candidate pairs by a user; Traversing the e-commerce transaction platform according to the second input to obtain a target product, where the target product refers to a product whose matching degree with the search intent of the first product is higher than a first threshold; The target product is displayed on the search interface according to the matching degree between the target product and the search intention of the first product.

2. The method according to claim 1, characterized in that: The first input is an initial text describing the product to be searched, and the method further includes: Extracting initial commodity key features of the initial text; Determine a first product feature of the product to be searched and a weight of each feature in the first product feature according to the initial product key feature; Inputting the first product feature and the weight of each feature in the first product feature into the query generation model to obtain a first search field set, wherein the first search field set includes a plurality of candidate search fields matching the first product feature; Determine a second product search intent according to the first search field set; Inputting the second product search intention into the generative model, obtaining first product description information and displaying it on the search interface; the first product description information includes product images of multiple candidate products and structured parameters of corresponding product features; receiving a third input from the user, where the third input indicates target product description information selected by the user from the first product description information; The e-commerce transaction platform is traversed based on the target product description information to obtain a plurality of first target products and are displayed on the search interface according to the correlation between each first target product and the first product feature.

3. The method according to claim 1 or 2, characterized in that: The first input is an initial image including a product to be searched, and the method comprises: Extracting image description information of the initial image; Determining a second product feature of the product to be searched according to the image description information; Inputting the second product feature and the initial image into a generative adversarial model to obtain a target image set, wherein the target image set includes a plurality of candidate product images associated with the second product feature; Displaying the target image set on the search interface; receiving a fifth input from the user, wherein the fifth input indicates a target product image selected by the user from the target image set; The e-commerce transaction platform is traversed based on the target product image to obtain a plurality of second target products and the second target products are displayed on the search interface according to the correlation between each second target product and the second product feature.

4. The method according to claim 3, characterized in that The step of inputting the second product feature and the initial image into the generative adversarial model to obtain a target image set includes: Performing feature dimension reduction on the initial image to obtain a candidate image; Noise of different intensities is added to the candidate images according to the second product feature to generate an initial image set. Denoising each candidate image in the initial image based on a diffusion model to obtain the target image set.

5. The method according to claim 3 or 4, characterized in that: After the target image set is displayed on the search interface, the method further includes: receiving a fifth input from a user on the target image set, wherein the fifth input includes commodity feature update information; In response to the fifth input, each candidate product image in the target image set is updated according to the product feature update information.

6. The method according to claim 2, characterized in that The step of traversing the e-commerce transaction platform based on the target product description information to obtain a plurality of first target products includes: Traversing the e-commerce trading platform to obtain product description information of candidate products, the product description information including product title, image features, page introduction parameters and user interaction features; The product description information is compared with the target product description information to obtain a plurality of first target products whose matching degree is higher than a second threshold.

7. The method according to claim 6, characterized in that Before inputting the initial commodity key features extracted based on the first input into the query generation model, the method further includes: When the first input is an initial text describing the product to be searched, if it is detected that the initial text includes adversarial noise, after performing intent recognition and entity extraction on the initial text, the initial text is subjected to text variation to obtain a target text; When the first input is an initial image including the product to be searched, backdoor detection is performed on the initial image. If it is detected that the initial image includes at least one trigger, denoising is performed on the detected at least one trigger.

8. The method according to claim 6, characterized in that The method further comprises: Add at least one trigger to the initial product image of the product to be distributed to obtain a target adversarial sample; Binding the target adversarial sample to the product page of the product to be distributed; Setting the target adversarial sample as the product image of the product to be distributed for distribution; The target adversarial sample is used to jump to the product page of the second product during a machine vision search.

9. A commodity search device, characterized in that: The large model quality evaluation device comprises: at least one processor and memory; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the method according to any one of claims 1 to 8.

10. A computer program product comprising instructions, the computer program product comprising program instructions, which, when executed on a computer or a processor, enable the computer or the processor to perform the method according to any one of claims 1 to 8.

Images