Image-based commodity searching method and device

Abstract

The embodiment of the invention discloses an image-based commodity search method and device, and the method comprises the steps: obtaining a commodity search image uploaded by a user, and extracting a commodity image feature vector of the commodity search image through a multi-modal feature extraction model; retrieving a pre-constructed image feature vector library by using the commodity image feature vector to obtain a first image feature vector similar to the commodity search image, and determining a first text feature vector having an index relationship with the first image feature vector according to the commodity identifier; retrieving a text feature vector library by using the commodity image feature vector to obtain a second text feature vector similar to the commodity search image, and determining a second image feature vector having an index relationship with the second text feature vector according to the commodity identifier; according to the method, the similarity between the commodity image feature vector and the commodity retrieval image-text pair is calculated, and the commodity search result is determined according to the similarity, so that the image search efficiency is improved, the image search precision is also improved, and the requirements of users are better met.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of Internet technology, and specifically to an image-based commodity search method and device. Background Art

[0002] Product search is a very important part of the e-commerce field, which directly affects the user's shopping experience and the merchant's sales performance. With the development of Internet technology, more and more product information is presented to users in the form of images, and image search technology has gradually become one of the important technologies in the e-commerce field. Existing image search technology mainly relies on models to extract image feature vectors and recall products through feature similarity.

[0003] Image search technology based on image feature vectors solves the problem of incomplete or inaccurate description of image content in traditional text-based search, but the technology still faces some challenges. First, the quality of product images varies. Due to reasons such as the shooting environment, the main body of the product image is unclear and the background contains interference information. Secondly, the description information of the product, such as title and detailed description, contains information that cannot be expressed in the image, such as size, color, material, etc. Relying solely on image search will ignore the correlation and complementarity between multimodal information, thereby limiting the further improvement of search results. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide an image-based product search method, apparatus, computing device, computer storage medium and computer program product that overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of an embodiment of the present application, there is provided an image-based commodity search method, comprising:

[0006] Obtain the product search image uploaded by the user, and extract the product image feature vector of the product search image using a pre-established multimodal feature extraction model;

[0007] Using the product image feature vector, a pre-built image feature vector library is retrieved to obtain a first image feature vector similar to the product search image, and a first text feature vector having an index relationship with the first image feature vector is determined according to the product identifier;

[0008] Using the product image feature vector to retrieve the text feature vector library to obtain a second text feature vector similar to the product search image, and determining the second image feature vector having an index relationship with the second text feature vector according to the product identification, wherein the first image feature vector and the first text feature vector constitute a product retrieval image-text pair, and the second text feature vector and the second image feature constitute a product retrieval image-text pair, the image feature vector library and the text feature vector library are respectively constructed by extracting features from all products using a pre-established multimodal feature extraction model, and the image feature vector library and the text feature vector library are associated through the product identification index;

[0009] Calculate the similarity between the product image feature vector and the product search image and text pair, and determine the product search results based on the similarity.

[0010] Furthermore, the image feature vectors in the image feature vector library and the text feature vectors in the text feature vector library are stored according to commodity categories;

[0011] After extracting the product image feature vector of the product search image using the pre-established multimodal feature extraction model, the method further includes: determining the product category to which the product to be searched belongs according to the product image feature vector;

[0012] Using the product image feature vector to retrieve a pre-built image feature vector library to obtain a first image feature vector similar to the product search image, and using the product image feature vector to retrieve a text feature vector library to obtain a second text feature vector similar to the product search image further includes:

[0013] The product image feature vector is used to retrieve the image feature vector under the product category in the image feature vector library to obtain a first image feature vector similar to the product search image. The product image feature vector is used to retrieve the text feature vector under the product category in the text feature vector library to obtain a second text feature vector similar to the product search image.

[0014] Furthermore, after obtaining the product search image uploaded by the user, the method further includes: determining image text description information corresponding to the product search image, and generating a product text feature vector according to the image text description information;

[0015] Using the product image feature vector to retrieve a pre-built image feature vector library to obtain a first image feature vector similar to the product search image, and using the product image feature vector to retrieve a text feature vector library to obtain a second text feature vector similar to the product search image further includes:

[0016] The product image feature vector and the product text feature vector are respectively used to retrieve a pre-constructed image feature vector library to obtain a first image feature vector similar to the product search image, and the product image feature vector and the product text feature vector are respectively used to retrieve a text feature vector library to obtain a second text feature vector similar to the product search image.

[0017] Furthermore, the training method of the multimodal feature extraction model includes:

[0018] Obtaining a product image-text pair training sample, wherein the product image-text pair training sample includes product images and product text data having the same product identifier;

[0019] The model is trained on the training samples according to the product pictures and texts to obtain a multimodal feature extraction model.

[0020] Furthermore, before performing model training on the training samples according to the images and texts to obtain the multimodal feature extraction model, the method further includes:

[0021] Performing image quality classification processing on the product images to obtain quality categories of the product images, and performing image filtering processing on the product images according to the quality categories;

[0022] Perform text filtering on product text data;

[0023] The model is trained on the training samples according to the pictures and texts, and the multimodal feature extraction model further includes:

[0024] The model is trained on the training samples according to the filtered product images and texts to obtain a multimodal feature extraction model.

[0025] Furthermore, the model training is performed on the training samples according to the filtered product images and texts to obtain the multimodal feature extraction model, which further includes:

[0026] Calculate the similarity between the product image and the product text data in the training sample after the filtered product image and text pair;

[0027] The training samples of product images and texts are selected according to the similarity, and the model is trained using the selected product images and texts to obtain a multimodal feature extraction model.

[0028] According to another aspect of an embodiment of the present application, there is provided an image-based commodity search device, comprising:

[0029] An acquisition module, adapted to acquire a product search image uploaded by a user;

[0030] An extraction module, adapted to extract a product image feature vector of a product search image using a pre-established multimodal feature extraction model;

[0031] A retrieval module is adapted to retrieve a pre-constructed image feature vector library using a product image feature vector to obtain a first image feature vector similar to a product search image, determine a first text feature vector having an index relationship with the first image feature vector according to a product identification, retrieve a text feature vector library using the product image feature vector to obtain a second text feature vector similar to the product search image, and determine a second image feature vector having an index relationship with the second text feature vector according to the product identification, wherein the first image feature vector and the first text feature vector constitute a product search image-text pair, and the second text feature vector and the second image feature constitute a product search image-text pair, the image feature vector library and the text feature vector library are respectively constructed by extracting features from all products using a pre-constructed multimodal feature extraction model, and the image feature vector library and the text feature vector library are associated through a product identification index;

[0032] The calculation module is suitable for calculating the similarity between the product image feature vector and the product search image and text pair, and determining the product search result according to the similarity.

[0033] According to another aspect of the embodiment of the present application, there is provided a computing device, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0034] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned image-based commodity search method.

[0035] According to another aspect of the embodiments of the present application, a computer storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned image-based product search method.

[0036] According to another aspect of the embodiments of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned image-based product search method.

[0037] According to the image-based product search method and device provided in the embodiments of the present application, a pre-established multimodal feature extraction model can be used to extract product image feature vectors of product search images uploaded by users, and then the product image feature vectors are used to retrieve a pre-built image feature vector library and a text feature vector library. The similarity between the product image feature vector and the product search image-text pair is calculated based on the search results, and the product search results are determined based on the similarity. This improves the image search efficiency on the one hand, and the image search accuracy on the other hand, thereby better meeting the needs of users.

[0038] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation method of the embodiment of the present application is listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the embodiments of the present application. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0040] Figure 1 A schematic diagram of a process of an image-based commodity search method according to an embodiment of the present application is shown;

[0041] Figure 2 A schematic diagram of processing product image and text pair training samples;

[0042] Figure 3 Schematic diagram of the process of the image-based product search method of Example 1;

[0043] Figure 4 Schematic diagram of the process of the image-based product search method of Example 2;

[0044] Figure 5 A structural block diagram of an image-based commodity search device according to an embodiment of the present application is shown;

[0045] Figure 6 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0047] The inventors of the present application have discovered that existing image search solutions only focus on image information and ignore other information. Therefore, many products that look similar but are very different will be recalled during the search. For example, searching with an image of a car will return a variety of car-related images, but the corresponding products may be toy cars, car accessories and other car-related categories. The corresponding product titles and other descriptive information can easily distinguish them, but traditional image searches ignore this part of the information. In order to solve the above problems, the inventors of the present application have proposed an image-based product search solution through creative work, which is explained below through specific examples:

[0048] Figure 1 A schematic diagram of a process flow of an image-based commodity search method according to an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:

[0049] Step S101: obtaining a product search image uploaded by a user, and extracting a product image feature vector of the product search image using a pre-established multimodal feature extraction model.

[0050] Specifically, when a user wants to use an image to search for a product, the user can upload a product search image through an upload function provided by the client, and the product search image includes the product that the user wants to search for this time.

[0051] The multimodal feature extraction model is trained using image-text pairs. Through this multimodal feature extraction model, image features and text features can be aligned in the same space. The multimodal feature extraction model can learn the intrinsic connection between images and texts, and realize cross-modal information fusion.

[0052] Therefore, after obtaining the product search image uploaded by the user, the pre-established multimodal feature extraction model can be used to extract the product image feature vector of the product search image, and the product image feature vector can reflect the visual features of the image.

[0053] Step S102: using the product image feature vector to retrieve a pre-built image feature vector library to obtain a first image feature vector similar to the product search image, and determining a first text feature vector having an index relationship with the first image feature vector according to the product identifier.

[0054] After establishing a multimodal feature extraction model, the present application needs to use the multimodal feature extraction model to extract features from the product data of all products in the entire site. The product data of each product includes product images and product text data. Therefore, the product images of all products in the entire site can be first imported into the multimodal feature extraction model, and the multimodal feature extraction model is used to extract image feature vectors from the product images. An image feature vector library is constructed using the extracted image feature vectors, wherein the image feature vector library at least stores in association with: product identification and corresponding image feature vectors; similarly, the product text data of all products in the entire site is imported into the multimodal feature extraction model, and the multimodal feature extraction model is used to extract text feature vectors from the product text data. A text feature vector library is constructed using the extracted text feature vectors, wherein the text feature vector library at least stores in association with: product identification and text feature vectors. The product text data mainly includes: product title, product category, product short description, etc. The product short description refers to a concise description of information such as product characteristics, functions, and uses. Among them, each image feature vector and text feature vector has a unique product identifier, so the image feature vector library and the text feature vector library can be associated through the product identifier, that is, the image feature vector library and the text feature vector library are associated through the product identifier (that is, product ID) as an index. Using the trained multimodal feature extraction model, the image feature vector and text feature vector of the product data of the entire site are extracted, and the image feature vector library and the text feature vector library are constructed respectively. The two vector libraries are associated through the product ID to provide data support for subsequent retrieval.

[0055] Therefore, after the product image feature vector is extracted using step S101, the product image feature vector can be used to retrieve a pre-constructed image feature vector library to obtain a first image feature vector similar to the product search image. Specifically, the similarity between the product image feature vector and the image feature vector in the image feature vector library can be calculated, and the first image feature vector similar to the product search image can be obtained by screening based on the similarity. For example, the similarity can be compared with a first similarity threshold. If the similarity between the product image feature vector and the image feature vector in the image feature vector library is greater than the first similarity threshold, the image feature vector is determined as the first image feature vector, wherein the first similarity threshold can be set based on experience.

[0056] After determining the first image feature vector, the product identifier associated with the first image feature vector can be determined, so that the product identifier can be used to query the text feature vector library to obtain the corresponding text feature vector associated with the product identifier, and the text feature vector can be determined as the first text feature vector that has an index relationship with the first image feature vector.

[0057] Step S103, using the product image feature vector to retrieve the text feature vector library to obtain a second text feature vector similar to the product search image, and determining the second image feature vector that has an index relationship with the second text feature vector according to the product identification, wherein the first image feature vector and the first text feature vector constitute a product retrieval image-text pair, and the second text feature vector and the second image feature constitute a product retrieval image-text pair, and the image feature vector library and the text feature vector library are respectively constructed by extracting features from all products using a pre-established multimodal feature extraction model, and the image feature vector library and the text feature vector library are associated through the product identification index.

[0058] After the product image feature vector is extracted using step S101, the product image feature vector can be used to retrieve a pre-constructed text feature vector library to obtain a second text feature vector similar to the product search image. Specifically, the similarity between the product image feature vector and the text feature vector in the text feature vector library can be calculated, and the second text feature vector similar to the product search image can be obtained by screening according to the similarity. For example, the similarity can be compared with a second similarity threshold. If the similarity between the product image feature vector and the text feature vector in the text feature vector library is greater than the second similarity threshold, the text feature vector is determined as the second text feature vector, wherein the second similarity threshold can be set based on experience, and the second similarity threshold can be the same as the first similarity threshold or different. Since the text feature vector library is constructed using a multimodal feature extraction model, and the product image feature vector is extracted using a multimodal feature extraction model, alignment between the product image feature vector and the text feature vector is achieved, so that similarity can be calculated.

[0059] After determining the second text feature vector, the product identifier associated with the second text feature vector can be determined, so that the product identifier can be used to query the image feature vector library to obtain the corresponding image feature vector associated with the product identifier, and the image feature vector can be determined as the second image feature vector that has an index relationship with the second text feature vector.

[0060] Through the above processing, it can be determined that the first image feature vector and the first text feature vector constitute a product retrieval image-text pair, and the second text feature vector and the second image feature vector constitute a product retrieval image-text pair.

[0061] In this step, there may be duplication between the product retrieval image-text pair composed of the first image feature vector and the first text feature vector and the product retrieval image-text pair composed of the second text feature vector and the second image feature. Therefore, the product retrieval image-text pairs may be deduplicated first, and then the deduplicated product retrieval image-text pairs may be used for subsequent processing.

[0062] Step S104, calculating the similarity between the product image feature vector and the product search image-text pair, and determining the product search result according to the similarity.

[0063] After determining the product retrieval image-text pair, it is necessary to calculate the similarity between the product image feature vector and the product retrieval image-text pair. For example, the similarity between the product image feature vector and the first text feature vector can be calculated. Combined with the calculated similarity between the product image feature vector and the first image feature vector, the similarity between the product image feature vector and the product retrieval image-text pair including the first text feature vector and the first image feature vector can be calculated; the similarity between the product image feature vector and the second image feature vector can be calculated. Combined with the calculated similarity between the product image feature vector and the second text feature vector, the similarity between the product image feature vector and the product retrieval image-text pair including the second text feature vector and the second image feature vector can be calculated. When calculating the similarity with the product retrieval image-text pair, a weighted method can be used, and the weight can be set according to actual needs.

[0064] After calculating the similarity between the product image feature vector and the product search image-text pair, filtering can be performed based on the similarity. For example, the product search image-text pairs whose similarity is less than or equal to the third similarity threshold are filtered out. The retained product search image-text pairs are the product search results. The corresponding image-text pairs can be displayed to the user to complete the product search. Alternatively, the K product search image-text pairs with the highest similarity can be retained. These K product search image-text pairs are the product search results.

[0065] Therefore, after the user uploads the image, the relevant products can be quickly retrieved from the image vector library and the text vector library. The similarity of image features and text features is considered simultaneously during the retrieval process, and the final retrieval result is obtained through comprehensive calculation.

[0066] There are many ways to calculate the above similarity, such as Euclidean distance, Manhattan distance, cosine distance, etc., which can be flexibly selected according to actual conditions.

[0067] In an optional implementation, in order to effectively reduce the retrieval of the feature vector library and improve the efficiency of product search, the image feature vectors in the image feature vector library and the text feature vectors in the text feature vector library can be classified and stored according to the product categories in advance, that is, the image feature vectors in the constructed image feature vector library are classified and stored according to the product categories, and the text feature vectors in the text feature vector library are also classified and stored according to the product categories. Then, when using the product image feature vector to search the image feature vector library and the text feature vector library, it is only necessary to search for specific product categories, thereby saving image search time.

[0068] Specifically, the product category to which the product to be searched belongs can be determined based on the product image feature vector. For example, each product category corresponds to a corresponding product category text label, and a feature vector corresponding to the product category text label is generated. The product category feature vector can be determined, and the similarity between the product image feature vector and each product category feature vector is calculated. The product category to which the product to be searched belongs is determined based on the similarity. For example, S product categories with the highest similarity can be selected as the product category to which the product to be searched belongs. Then, under the limited product category, the product image feature vector is used to retrieve the image feature vector in the image feature vector library to obtain a first image feature vector similar to the product search image, and the product image feature vector is used to retrieve the text feature vector in the text feature vector library to obtain a second text feature vector similar to the product search image.

[0069] In an optional implementation, after obtaining the product search image uploaded by the user, the image text description information corresponding to the product search image can also be determined, and the product text feature vector can be generated based on the image text description information. For example, the image labeling model can be used to process the product search image to determine the corresponding image text description information. Then, the product image feature vector and the product text feature vector can be used to retrieve a pre-built image feature vector library to obtain a first image feature vector similar to the product search image. The product image feature vector and the product text feature vector can be used to retrieve a text feature vector library to obtain a second text feature vector similar to the product search image. The specific implementation process is similar to the above and will not be repeated here.

[0070] In an optional implementation, the following method is used to train the multimodal feature extraction model:

[0071] Obtain product image-text pair training samples;

[0072] The model is trained on the training samples according to the product pictures and texts to obtain a multimodal feature extraction model.

[0073] Specifically, commodity data usually includes commodity images and commodity text data, and thus, commodity images and commodity text data can form an image-text pair. Therefore, a commodity image-text pair training sample includes commodity images and commodity text data with the same commodity identification. A large number of commodity image-text pair training samples can be obtained here, and then, the model is trained according to the obtained commodity image-text pair training samples. For example, the Clip model or the Blip model can be used for training. Of course, other models can also be used. There is no specific limitation here. A multimodal feature extraction model can be obtained through training. By training a multimodal feature extraction model (such as Clip, Blip, etc.), image features and text features can be aligned in the same space to achieve cross-modal information fusion. The training data of the model needs to contain rich image-text pairs to ensure that the model can learn the intrinsic connection between images and texts.

[0074] In order to make full use of various product description information, the product title, product category, product short description, etc. can be combined with the product image to form a picture-text pair for training, which greatly increases the training data. Since the picture-text pairs are used during training, the training goal is to make a certain product image have a low similarity with the product text data that constitutes the picture-text pair, while with other product text data that do not constitute a picture-text pair. In this way, the features of different modalities can be unified into one space. For example, the image of clothes and the text of clothes are unified into an abstract concept of clothes, thereby unifying different modalities from heterogeneity into one space and realizing information fusion between different modalities.

[0075] Here, a multimodal model is used to align image features and text features instead of two independent feature extraction networks. This can solve the problem of data heterogeneity between different modal data, making it easier to calculate the similarity between different modal data in the future.

[0076] In addition, although the training data is sufficient, the quality of many product images is very poor due to various reasons, or the product text data contains some meaningless product titles or product descriptions. In order to improve the accuracy of the multimodal feature extraction model, the product image and text training samples can be processed before model training. Figure 2A schematic diagram of processing training samples of product images and text pairs is provided. Specifically, the training samples of product images and text pairs may be obtained first, and then the product images may be subjected to image quality classification processing to obtain the quality categories of the product images. For example, a pre-trained image quality classification model may be used to perform image quality classification processing on the product images, wherein the image quality classification model may be trained using the following method: obtaining high-quality and low-quality product image samples, and performing quality annotation on each product image sample, for example, 1 represents high quality and 0 represents low quality, and using the product image samples and the quality annotation results to perform model training to obtain an image quality classification model, and performing image filtering processing on the product images according to the quality categories to filter out product images with poor image quality. For example, if a product image is judged to be of “low quality”, Then it can be filtered out from the training sample set, where a product image of a product may contain a main image and multiple sub-images. When performing image quality classification processing, each image needs to be processed; then the product text data is subjected to text filtering processing, for example, a filter text set is pre-set, and some meaningless product titles, product short descriptions or other meaningless words are stored in the filter text set, such as advertising slogans, etc. The product text data can be matched with the pre-set filter text set. If a certain product text data contains advertising slogans or other meaningless product titles or product short descriptions, it can be filtered out from the training sample set, and finally the model is trained based on the filtered product image and text training samples to obtain a multimodal feature extraction model. Among them, the product image and text training sample refers to a data set containing product images and product text data with the same product identification. The quality category of the product image reflects the image quality, such as clarity, such as low quality, high quality, etc. The product text data refers to data such as product titles, product categories, and product short descriptions.

[0077] It should be noted that the image quality classification process and the text filtering process may be performed simultaneously or sequentially, and the specific execution order is not specifically limited here.

[0078] In order to further improve the accuracy of the multimodal feature extraction model, the product image-text pair training samples can be screened before model training. Specifically, after completing the product image filtering and product text data filtering, the similarity between the product image and the product text data in the filtered product image-text pair training samples can be calculated first, and then the product image-text pair training samples can be screened according to the similarity. For example, the product image-text pair training samples with a similarity higher than the fourth similarity threshold can be screened out, and the rest can be discarded, which solves the problem of mismatch between the product image and the product text data of the same product. Finally, the screened product image-text pair training samples are used for model training to obtain a multimodal feature extraction model. Among them, the higher the similarity, the more the product image and the product text data match. Therefore, when performing model training, those product image-text pair training samples with high similarity should be given priority, which improves the data quality of the training samples. Usually, a product image of a product may include a main image and multiple sub-images. Therefore, the similarity between the main image and the product text data can be calculated separately, and the similarity between each sub-image and the product text data can be calculated. A weighted calculation is performed based on the calculated similarities to obtain the similarity between the product image and the product text data. Alternatively, an open source model (a public model) can be used to perform a similarity calculation on all product images and product text data of a product, and the combinations with low similarity can be filtered out, thereby improving the quality of the data.

[0079] The following is a specific example to illustrate the implementation process of image-based product search:

[0080] Figure 3 FIG. 1 is a flowchart of an image-based product search method of Example 1. Figure 3 As shown, (1) the user can upload a product search image, and firstly use the multimodal feature extraction model to extract the product image feature vector of the product search image;

[0081] (2) Based on the product image feature vector, similarity calculation is used to obtain S product categories to which the searched product may belong. For example, product categories are obtained based on multimodality. Each product category corresponds to a corresponding product category text label. A feature vector corresponding to the product category text label is generated, thereby determining the product category feature vector. The similarity between the product image feature vector and each product category feature vector is calculated. Multimodality here refers to calculating the similarity between the product image feature vector and the product category feature vector corresponding to the product category text label to obtain the product category.

[0082] (3) Under the determined limited product category, the product image feature vector is used to search the image feature vector library and the text feature vector library respectively, wherein the results can be filtered according to the similarity threshold. Specifically, M image feature vectors and N text feature vectors are obtained.

[0083] (4) The text feature vector library is searched according to the product ID corresponding to the M image feature vectors to obtain the corresponding M text feature vectors. The image feature vector library is searched according to the product ID corresponding to the N text feature vectors to obtain the corresponding N image feature vectors. At this time, N+M image-text pairs can be obtained.

[0084] (5) Calculate the similarity between the product image feature vector and the M text feature vectors and N image feature vectors.

[0085] (6) The similarity scores between the product image feature vector and the M+N product image-text pairs are calculated through a comprehensive formula, and then filtered through a similarity threshold to obtain the final K products as the result.

[0086] Figure 4 FIG. 1 is a flowchart of the image-based product search method of Example 2. Figure 4 As shown, (1) a user can upload a product search image, firstly use the multimodal feature extraction model to extract the product image feature vector of the product search image, determine the image text description information corresponding to the product search image, and generate the product text feature vector according to the image text description information;

[0087] (2) Using similarity calculation based on the product image feature vector, S product categories to which the searched product may belong are obtained, for example, product categories are obtained based on multimodality.

[0088] (3) Under the determined limited product category, the product image feature vector and the product text feature vector are used to search in the image feature vector library and the text feature vector library respectively, wherein the results can be filtered according to the similarity threshold. Specifically, searching using the product image feature vector will obtain M image feature vectors and N text feature vectors, and searching using the product text feature vector will obtain M image feature vectors and N text feature vectors.

[0089] (4) Search the text feature vector library according to the product ID corresponding to the 2M image feature vectors to obtain the corresponding 2M text feature vectors. Search the image feature vector library according to the product ID corresponding to the 2N text feature vectors to obtain the corresponding 2N image feature vectors. At this time, 2M+2N image-text pairs can be obtained.

[0090] (5) Calculate the similarity between the product image feature vector and the 2M text feature vectors and 2N image feature vectors.

[0091] (6) The similarity scores between the product image feature vector and the 2M+2N product image-text pairs are calculated through a comprehensive formula, and then filtered through a similarity threshold to obtain the final K products as the result.

[0092] In an optional embodiment, the method also includes: obtaining a product image feature vector and a product text feature vector of the product corresponding to the user's historical behavior, wherein the user's historical behavior includes historical product purchasing behavior and historical product browsing behavior, using the product image feature vector and the product text feature vector to retrieve a pre-constructed image feature vector library to obtain a similar third image feature vector, determining a third text feature vector that has an index relationship with the third image feature vector based on the product identification, using the product image feature vector and the product text feature vector to retrieve a text feature vector library to obtain a fourth text feature vector that is similar to the product search image, determining a fourth image feature vector that has an index relationship with the fourth text feature vector based on the product identification, and recommending the product corresponding to the third image feature vector and the third text feature vector and the product corresponding to the fourth image feature vector and the fourth text feature vector to the user.

[0093] During the model training process, this application uses a variety of data processing and filtering methods to ensure the quality of training data. Since category data is used for training, category restrictions can also be imposed on products. For example, if an image of a car is used for retrieval, the recalled similarities all include cars. However, if there is a model picture in the clothing category with the same car in the background, there will be a high degree of similarity, so the recommended product may be clothing. However, what the user really wants to search for is car-related products. After using category restrictions, the search category is first restricted to cars and car-related categories based on the image. In this way, the search will not result in clothing. By introducing category information, the search scope is reduced, thereby improving the accuracy of the search.

[0094] The use of multimodal feature extraction models for feature extraction and use brings unprecedented flexibility and accuracy to product retrieval. By combining the features of images and text, it can not only capture the appearance information of the product, but also deeply understand the description and attributes behind it, thereby achieving a more comprehensive and accurate understanding of the product. This cross-modal information fusion enables the product retrieval system to be closer to the real needs of users and provide more intimate and personalized services.

[0095] In addition, the application of multimodal features has greatly improved the search efficiency. In traditional single-modal search, due to the limitation of information dimension, users often need to perform multiple queries and screenings to find satisfactory products. Multimodal features, by considering both image and text information, achieve full coverage of product information, thereby reducing the number of user queries and improving the accuracy and efficiency of search.

[0096] Finally, the application of multimodal features also brings more innovative possibilities to the system. For example, multimodal features can be used to recommend products. By analyzing the user's browsing history and purchasing behavior, combined with the image and text features of the product, products that better suit the user's interests and needs can be recommended to the user. This recommendation algorithm based on multimodal features can not only improve the accuracy of recommendations, but also bring users a richer shopping experience.

[0097] The image-based product search method provided in the embodiment of the present application can use a pre-established multimodal feature extraction model to extract the product image feature vector of the product search image uploaded by the user, and then use the product image feature vector to retrieve a pre-built image feature vector library and a text feature vector library, calculate the similarity between the product image feature vector and the product search image-text pair based on the search results, and determine the product search results based on the similarity. On the one hand, the image search efficiency is improved, and on the other hand, the dual recall of image and text features is performed simultaneously, which also improves the image search accuracy, thereby better meeting the needs of users.

[0098] Figure 5 FIG. 4 shows a structural block diagram of an image-based commodity search device according to an embodiment of the present application. Figure 5 As shown, the device comprises:

[0099] An acquisition module 501 is adapted to acquire a product search image uploaded by a user;

[0100] An extraction module 502, adapted to extract a product image feature vector of a product search image using a pre-established multimodal feature extraction model;

[0101] A retrieval module 503 is adapted to retrieve a pre-constructed image feature vector library using a product image feature vector to obtain a first image feature vector similar to the product search image, determine a first text feature vector having an index relationship with the first image feature vector according to the product identification, retrieve a text feature vector library using the product image feature vector to obtain a second text feature vector similar to the product search image, and determine a second image feature vector having an index relationship with the second text feature vector according to the product identification, wherein the first image feature vector and the first text feature vector constitute a product search image-text pair, and the second text feature vector and the second image feature constitute a product search image-text pair, the image feature vector library and the text feature vector library are respectively constructed by extracting features from all products using a pre-constructed multimodal feature extraction model, and the image feature vector library and the text feature vector library are associated through a product identification index;

[0102] The calculation module 504 is adapted to calculate the similarity between the product image feature vector and the product search image-text pair, and determine the product search result according to the similarity.

[0103] Optionally, the image feature vectors in the image feature vector library and the text feature vectors in the text feature vector library are stored according to commodity categories;

[0104] The device further comprises: a determination module adapted to determine the commodity category to which the commodity to be searched belongs according to the commodity image feature vector;

[0105] The retrieval module is further adapted to: using the product image feature vector to retrieve the image feature vector under the product category in the image feature vector library to obtain a first image feature vector similar to the product search image; using the product image feature vector to retrieve the text feature vector under the product category in the text feature vector library to obtain a second text feature vector similar to the product search image.

[0106] Optionally, the device further comprises: a generating module adapted to determine image text description information corresponding to the commodity search image, and generate a commodity text feature vector according to the image text description information;

[0107] The retrieval module is further adapted to: respectively use the product image feature vector and the product text feature vector to retrieve a pre-constructed image feature vector library to obtain a first image feature vector similar to the product search image, and respectively use the product image feature vector and the product text feature vector to retrieve a text feature vector library to obtain a second text feature vector similar to the product search image.

[0108] Optionally, the device further comprises: a multimodal feature extraction model training module, adapted to obtain product image-text pair training samples, wherein the product image-text pair training samples comprise product images and product text data having the same product identifier;

[0109] The model is trained on the training samples according to the product pictures and texts to obtain a multimodal feature extraction model.

[0110] Optionally, the multimodal feature extraction model training module is further adapted to: perform image quality classification processing on the commodity images to obtain quality categories of the commodity images, and perform image filtering processing on the commodity images according to the quality categories;

[0111] Perform text filtering on product text data;

[0112] The model is trained on the training samples according to the filtered product images and texts to obtain a multimodal feature extraction model.

[0113] Optionally, the multimodal feature extraction model training module is further adapted to: calculate the similarity between the product image and the product text data in the filtered product image-text pair training sample;

[0114] The training samples of product images and texts are selected according to the similarity, and the model is trained using the selected product images and texts to obtain a multimodal feature extraction model.

[0115] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0116] The image-based product search device provided in the embodiment of the present application can use a pre-established multimodal feature extraction model to extract the product image feature vector of the product search image uploaded by the user, and then use the product image feature vector to retrieve a pre-built image feature vector library and a text feature vector library, calculate the similarity between the product image feature vector and the product search image-text pair based on the search results, and determine the product search results based on the similarity, which improves the image search efficiency on the one hand, and improves the image search accuracy on the other hand, thereby better meeting the needs of users.

[0117] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program, which can enable a processor to perform operations corresponding to the image-based product search method in any of the above method embodiments.

[0118] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform operations corresponding to the image-based product search method in any of the above method embodiments.

[0119] Figure 6 A schematic diagram of the structure of an embodiment of a computing device of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0120] like Figure 6 As shown, the computing device may include: a processor (processor) 602 , a communications interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0121] The processor 602, the communication interface 604, and the memory 606 communicate with each other via a communication bus 608. The communication interface 604 is used to communicate with other devices such as a client or a network element of another server. The processor 602 is used to execute a program 610, which can specifically execute the relevant steps in the above-mentioned embodiment of the image-based commodity search method for a computing device.

[0122] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0123] The processor 602 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0124] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0125] The program 610 can be specifically used to enable the processor 602 to execute the image-based commodity search method in any of the above-mentioned method embodiments. The specific implementation of each step in the program 610 can refer to the corresponding descriptions in the corresponding steps and units in the above-mentioned image-based commodity search embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0126] The algorithm and display provided herein are not inherently related to any specific computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application embodiment described herein, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application embodiment.

[0127] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0128] Similarly, it should be understood that in order to streamline the present disclosure and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed embodiments of the present application require more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the embodiment of the present application.

[0129] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0130] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the embodiments of the present application and to form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0131] The various component embodiments of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0132] It should be noted that the above embodiments illustrate rather than limit the embodiments of the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The embodiments of the present application may be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A product search method based on an image, comprising: Acquire a product search image uploaded by a user, and extract a product image feature vector of the product search image using a pre-established multimodal feature extraction model; Using the product image feature vector, searching a pre-built image feature vector library to obtain a first image feature vector similar to the product search image, and determining a first text feature vector having an index relationship with the first image feature vector according to the product identifier; Using the product image feature vector to retrieve a text feature vector library to obtain a second text feature vector similar to the product search image, and determining a second image feature vector having an index relationship with the second text feature vector according to the product identification, wherein the first image feature vector and the first text feature vector constitute a product search image-text pair, and the second text feature vector and the second image feature constitute a product search image-text pair, the image feature vector library and the text feature vector library are respectively constructed by extracting features from all products using a pre-established multimodal feature extraction model, and the image feature vector library and the text feature vector library are associated with each other through a product identification index; The similarity between the product image feature vector and the product search image-text pair is calculated, and the product search result is determined according to the similarity.

2. The method according to claim 1, wherein: The image feature vectors in the image feature vector library and the text feature vectors in the text feature vector library are stored according to commodity categories; After extracting the product image feature vector of the product search image using the pre-established multimodal feature extraction model, the method further includes: determining the product category to which the product to be searched belongs according to the product image feature vector; The step of using the product image feature vector to retrieve a pre-built image feature vector library to obtain a first image feature vector similar to the product search image, and using the product image feature vector to retrieve a text feature vector library to obtain a second text feature vector similar to the product search image further comprises: The product image feature vector is used to retrieve the image feature vector under the product category in the image feature vector library to obtain a first image feature vector similar to the product search image. The product image feature vector is used to retrieve the text feature vector under the product category in the text feature vector library to obtain a second text feature vector similar to the product search image.

3. The method according to claim 1 or 2, wherein: After obtaining the product search image uploaded by the user, the method further includes: determining image text description information corresponding to the product search image, and generating a product text feature vector according to the image text description information; The step of using the product image feature vector to retrieve a pre-built image feature vector library to obtain a first image feature vector similar to the product search image, and using the product image feature vector to retrieve a text feature vector library to obtain a second text feature vector similar to the product search image further comprises: The product image feature vector and the product text feature vector are respectively used to search a pre-constructed image feature vector library to obtain a first image feature vector similar to the product search image, and the product image feature vector and the product text feature vector are respectively used to search a text feature vector library to obtain a second text feature vector similar to the product search image.

4. The method according to claim 1 or 2, wherein: The training method of the multimodal feature extraction model includes: Acquire a product image-text pair training sample, wherein the product image-text pair training sample includes product images and product text data having the same product identifier; Model training is performed on training samples according to the product images and texts to obtain a multimodal feature extraction model.

5. The method according to claim 4, wherein: Before performing model training on the training samples according to the images and texts to obtain a multimodal feature extraction model, the method further includes: Performing image quality classification processing on the commodity image to obtain a quality category of the commodity image, and performing image filtering processing on the commodity image according to the quality category; Performing text filtering processing on the commodity text data; The step of performing model training on the training samples according to the images and texts to obtain a multimodal feature extraction model further includes: The model is trained on the training samples according to the filtered product images and texts to obtain a multimodal feature extraction model.

6. The method according to claim 5, wherein: The step of performing model training on the training samples according to the filtered product images and texts to obtain a multimodal feature extraction model further includes: Calculate the similarity between the product image and the product text data in the training sample after the filtered product image and text pair; The training samples of product images and texts are selected according to the similarity, and the model is trained using the selected product images and texts to obtain a multimodal feature extraction model.

7. A commodity search device based on an image, comprising: An acquisition module, adapted to acquire a product search image uploaded by a user; An extraction module, adapted to extract a product image feature vector of the product search image using a pre-established multimodal feature extraction model; A retrieval module, adapted to use the product image feature vector to retrieve a pre-constructed image feature vector library to obtain a first image feature vector similar to the product search image, determine a first text feature vector having an index relationship with the first image feature vector according to a product identification, use the product image feature vector to retrieve a text feature vector library to obtain a second text feature vector similar to the product search image, and determine a second image feature vector having an index relationship with the second text feature vector according to the product identification, wherein the first image feature vector and the first text feature vector constitute a product retrieval image-text pair, and the second text feature vector and the second image feature constitute a product retrieval image-text pair, the image feature vector library and the text feature vector library are respectively constructed by extracting features from all products using a pre-constructed multimodal feature extraction model, and the image feature vector library and the text feature vector library are associated through a product identification index; The calculation module is adapted to calculate the similarity between the product image feature vector and the product search image-text pair, and determine the product search result according to the similarity.

8. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the image-based product search method according to any one of claims 1 to 6.

9. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute operations corresponding to the image-based product search method according to any one of claims 1 to 6.

10. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the image-based product search method according to any one of claims 1 to 6.

Images