Commodity searching method and system based on multiple modes

Abstract

The invention provides a commodity searching method and system based on multiple modes, and relates to the technical field of commodity searching. According to the specific implementation scheme, graph retrieval and image-text retrieval are carried out in a plurality of first candidate commodities based on a graph vector of a query graph to obtain a first comprehensive score of the query graph and each first candidate commodity, and if the score does not meet a preset scoring condition, semantic analysis is carried out on the query graph to obtain a text description vector of the query graph; and performing text-graph and text-text retrieval in the plurality of second candidate commodities based on the text description vector of the query graph to obtain a second comprehensive score of the query graph and each second candidate commodity, and sorting each second candidate commodity based on the score to obtain a commodity search result of the query graph. By adopting the method and the device, the accuracy of graph-based retrieval can be improved.

Description

Technical Field

[0001] The present invention relates to the field of information search technology, and in particular to a commodity search method and system based on multimodality. Background Art

[0002] In the product search of e-commerce platforms, the retrieval method is generally based on text. Users enter keywords in the system, and the system searches for products based on the match between the keywords and the product text description. However, this method has some limitations: 1. Inaccurate expression. It may be difficult for users to describe the appearance, features and other details of the product with accurate words, resulting in inaccurate search results. For example, for a fashion product with a unique design, if the user may not be able to accurately describe its appearance characteristics with appropriate words, the relevant product may not be retrieved.

[0003] 2. Difficulty in semantic understanding. It is difficult for the system to fully understand the complex semantics behind the keywords entered by users. The same word may have multiple meanings, and different users may use different words to describe the same product, which may cause the system to deviate when matching products.

[0004] Therefore, in order to solve the shortcomings of traditional text keyword search, the image search technology came into being. It allows users to search for products by uploading product pictures, which is more intuitive and convenient. Users do not need to rack their brains to think about how to describe the product in words. They only need to provide a picture of the product, and the system can try to identify the product in the picture and find relevant product information. However, this method also has the following problems: 1. Inaccurate retrieval of the same or similar products. Although the existing image recognition and product search methods have their advantages, they also have some technical problems. For example, most systems rely only on image feature similarity retrieval, extract image features through pre-trained neural network models, and then obtain score rankings based on feature similarity retrieval. However, due to the uncontrollability of vector matching, only visually similar image rankings can be obtained, and there is no guarantee that the same or similar products can be retrieved.

[0005] 2. Insufficient use of multi-view images. Products usually have images from multiple viewpoints, but existing systems do not fully utilize these multi-view images. Some systems simply process single-view images and do not establish an effective mechanism to comprehensively utilize the information of multiple viewpoint images, resulting in incomplete and inaccurate search results.

[0006] 3. Poor retrieval effect of special product images. When users input product images that do not exist in the mall or have a large appearance difference, the product ranking retrieved by the existing system often cannot meet customer needs. Due to the lack of an effective processing mechanism, the search results usually have a large number of category deviations and visual deviations.

[0007] 4. Low efficiency of inference model calling. In actual applications of e-commerce platforms, there may be a situation where a large number of users are performing image recognition and product search operations at the same time. However, the existing system has not been effectively optimized in inference model calling and cannot meet the high-concurrency query requirements, resulting in long waiting times for users. Summary of the invention

[0008] The present invention provides a multimodal commodity search method and system, which can solve at least one of the above technical problems.

[0009] According to one aspect of the present invention, there is provided a commodity search method, comprising: Based on the graph vector of the query graph and the graph vectors of each perspective graph of each first candidate commodity, determining the graph-to-graph similarity between the query graph and each of the first candidate commodities; Based on the graph vector of the query graph and the text description vector of each of the first candidate products, determining the graph and text similarity between the query graph and each of the first candidate products; Determine a first comprehensive score for the query graph and each of the first candidate products based on the image-image similarity and image-text similarity between the query graph and each of the first candidate products; When the query graph and the first comprehensive scores of the first candidate products do not meet a preset score condition, semantically analyze the query graph to obtain a text description vector of the query graph; Determine the text and image similarity between the query graph and each of the second candidate products based on the text description vector of the query graph and the image vector of each of the perspective images of each of the second candidate products; Determining text similarity between the query graph and each of the second candidate commodities based on the text description vector of the query graph and the text description vector of each of the second candidate commodities; Determine a second comprehensive score for the query graph and each of the second candidate products based on the text-to-image similarity and text-to-text similarity between the query graph and each of the second candidate products; Based on the query graph and the second comprehensive scores of the second candidate commodities, the second candidate commodities are sorted to obtain commodity search results of the query graph.

[0010] According to another aspect of the present invention, there is provided a commodity search device, comprising: A first similarity calculation module, configured to determine the image-image similarity between the query image and each of the first candidate products based on the image vector of the query image and the image vector of each of the perspective images of each of the first candidate products; a second similarity calculation module, configured to determine the image and text similarity between the query graph and each of the first candidate products based on the graph vector of the query graph and the text description vector of each of the first candidate products; A first scoring module, configured to determine a first comprehensive score of the query graph and each of the first candidate commodities based on a graph-graph similarity and a graph-text similarity between the query graph and each of the first candidate commodities; A semantic parsing module, configured to perform semantic parsing on the query graph to obtain a text description vector of the query graph when the query graph and the first comprehensive scores of each of the first candidate products do not meet a preset score condition; A third similarity calculation module, configured to determine the text and image similarity between the query graph and each of the second candidate products based on the text description vector of the query graph and the image vector of each of the perspective images of each of the second candidate products; a fourth similarity calculation module, configured to determine the text similarity between the query graph and each of the second candidate commodities based on the text description vector of the query graph and the text description vector of each of the second candidate commodities; A second scoring module, configured to determine a second comprehensive score of the query graph and each of the second candidate commodities based on a text-to-picture similarity and a text-to-text similarity between the query graph and each of the second candidate commodities; The first result determination module is used to sort each of the second candidate commodities based on the query graph and the second comprehensive score of each of the second candidate commodities to obtain a commodity search result of the query graph.

[0011] According to another aspect of the present invention, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any commodity search method in the embodiments of the present invention.

[0012] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any commodity search method according to the embodiments of the present invention.

[0013] By adopting the technical solution of the present invention, based on the graph vector of the query graph and the graph vector of each perspective graph of each first candidate commodity, the graph-to-graph similarity between the query graph and each first candidate commodity is determined; based on the graph vector of the query graph and the text description vector of each first candidate commodity, the graph-to-text similarity between the query graph and each first candidate commodity is determined; based on the graph-to-graph similarity and the graph-to-text similarity between the query graph and each first candidate commodity, the first comprehensive score between the query graph and each first candidate commodity is determined. In this way, the input query graph can be used to evaluate the graph-to-graph similarity and the graph-to-text similarity between the query graph and the candidate commodity, thereby achieving the effect of graph retrieval. In the case where the first comprehensive score of the query graph and each first candidate product does not meet the preset scoring conditions, the query graph is semantically parsed to obtain the text description vector of the query graph; based on the text description vector of the query graph and the graph vector of each perspective graph of each second candidate product, the text-to-image similarity between the query graph and each second candidate product is determined; based on the text description vector of the query graph and the text description vector of each second candidate product, the text-to-text similarity between the query graph and each second candidate product is determined; based on the text-to-image similarity and text-to-text similarity between the query graph and each second candidate product, the second comprehensive score of the query graph and each second candidate product is determined; based on the second comprehensive score of the query graph and each second candidate product, the target candidate product is determined among the second candidate products. In this way, when the graph retrieval is not ideal, the query graph can be semantically parsed to obtain the text description information of the query graph, so that the text description information of the query graph can be used for graph-text retrieval to obtain the corresponding retrieval results. Therefore, the technical solution of the present invention can improve the accuracy of graph retrieval.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention. Figure 1 is a flow chart of a commodity search method according to an embodiment of the present invention; Figure 2 is a flow chart of a commodity search method according to another embodiment of the present invention; Figure 3 is a flow chart of a commodity search method according to another embodiment of the present invention; Figure 4 is a structural block diagram of a commodity search device according to an embodiment of the present invention; Figure 5 The block diagram is a block diagram of an electronic device for implementing the method according to the embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present invention. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0017] Explanation of terms Multimodality: involves multiple data modalities, such as text, images, audio, etc. The data of these modalities can be related and complemented to each other for a more comprehensive description and understanding of information.

[0018] Pre-trained model: A deep learning model that is pre-trained on large-scale data. The features and patterns learned on a large amount of general data can be fine-tuned or used directly in subsequent specific tasks to improve the performance of the model on the task.

[0019] CLIP (Contrastive Language-Image Pretraining) is a multimodal learning model that aims to associate images and texts. It can quickly perform tasks such as image-text feature similarity calculation, cross-modal retrieval, and zero-shot image classification.

[0020] Chinese-CLIP: Since the original CLIP model is based on English text and image corpus, it cannot be used for Chinese text and image representation extraction scenarios. Therefore, Chinese-CLIP implements a Chinese version of the CLIP model based on large-scale native Chinese text and image data to meet the demand for a Chinese version.

[0021] Image search: a product search method based on image recognition technology. Users upload or provide pictures of products, and the system can identify the products in the pictures and search for related product information in the product database of the e-commerce platform, including the same style and similar products.

[0022] Large model: A high-performance, high-computation deep neural network model trained with large-scale training data through pre-training, fine-tuning, instruction fine-tuning, etc.

[0023] Figure 1 It is a flow chart of a commodity search method according to an embodiment of the present invention.

[0024] like Figure 1 As shown, the product search method may include: S110, determining the graph-to-graph similarity between the query graph and each first candidate product based on the graph vector of the query graph and the graph vector of each view graph of each first candidate product; S120, determining the image and text similarity between the query graph and each first candidate product based on the graph vector of the query graph and the text description vector of each first candidate product; S130, determining a first comprehensive score of the query graph and each first candidate product based on the image-image similarity and image-text similarity between the query graph and each first candidate product; S140, when the query graph and the first comprehensive scores of each first candidate product do not meet the preset score condition, semantically analyze the query graph to obtain a text description vector of the query graph; S150, determining the text and image similarity between the query graph and each second candidate product based on the text description vector of the query graph and the image vector of each view image of each second candidate product; S160, determining the text similarity between the query graph and each second candidate commodity based on the text description vector of the query graph and the text description vector of each second candidate commodity; S170, determining a second comprehensive score of the query graph and each second candidate product based on the text-to-image similarity and text-to-text similarity between the query graph and each second candidate product; S180, sorting the second candidate commodities based on the query graph and the second comprehensive scores of the second candidate commodities to obtain commodity search results of the query graph.

[0025] Exemplarily, the commodity search method can be applied to text or product search. For example, a query graph is input into an e-commerce platform, and the platform searches for commodities according to the query graph and displays search results, which may be commodity graphs of one or more commodities, corresponding commodity summaries, and commodity detail links.

[0026] For example, an image encoding module (Chinese-CLIP model) can be used to encode the query graph and the view graphs of each candidate product to convert the image into a vector form, which is convenient for subsequent similarity calculation. This module is the basis of the entire system and provides image vector data for other modules.

[0027] It is understandable that a candidate product may have multiple appearance images from different perspectives, namely the above-mentioned perspective images. The first candidate product and the second candidate product are any products in the candidate product library, and the first candidate product and the second candidate product may be the same or different.

[0028] For example, a vector database module may be used to store the image vectors of the view images of each candidate product and the associated information with the product name description. Figure 2 As shown, the identification information and vectors of candidate products are mapped and stored one-to-many to support multi-view product image comparison. This module plays the role of data storage and search support in the system.

[0029] Exemplarily, cosine similarity may be used to calculate the image-text similarity between the image vector of the query image and the view image of the candidate product.

[0030] For example, Figure 3 As shown, using the Chinese-CLIP model, the input query graph Q and product graph ( represents the jth product image of the ith product) and encodes them respectively to obtain the query graph vector and product image vector Then, the similarity between the query graph vector and each product graph vector is calculated using the cosine similarity formula: .

[0031] Exemplarily, the vector database module may store text description vectors of candidate commodities. The vector is a vector representation of text description information of the candidate commodities. The text description information of the candidate commodities may include association information between the image vector of the candidate commodities and the commodity name.

[0032] For example, product name Encoded into product name description vector through Chinese-CLIP model . Compute the query graph vector Vector with product name description The similarity between them also uses the cosine similarity formula: .

[0033] Exemplarily, a weighted sum is performed on the image-image similarity and the image-text similarity between the query image and the first candidate product to obtain a first comprehensive score of the query image and the first candidate product.

[0034] For example, Figure 3 As shown, since the product graph vector { } will have a many-to-one relationship with the product identifier, and they are generally feature vectors of different angles of the same product, so we use Get the score of the graph closest to the query graph, that is, the similarity between the above graphs.

[0035] Then, the image-image matching score (image-image similarity) and image-text matching score (image-text similarity) are combined.

[0036] Using the weighted summation method, the weight of the graph-graph matching score is set to , the weight of the image-text matching score is , the comprehensive score is: ; ; .

[0037] Finally, according to the comprehensive score The products in the library are ranked, and the products with higher scores are ranked higher in the search results.

[0038] For example, the scoring condition may be that the first comprehensive scores of the query graph and each first candidate product are all less than a preset scoring threshold, or that the maximum value of the first comprehensive scores of the query graph and each first candidate product is less than a preset scoring threshold.

[0039] It can be understood that the query graph and the first comprehensive scores of the first candidate products do not meet the preset scoring conditions, which means that the graph retrieval effect of the query graph is not ideal.

[0040] Exemplarily, a pre-trained large language model may be used to perform semantic analysis on the query graph to obtain a text description vector of the query graph.

[0041] Exemplarily, cosine similarity may be used to calculate the distance between the text description vector of the query graph and the image vectors of each view image of the second candidate product, thereby obtaining the text and image similarity between the query graph and the second candidate product.

[0042] Exemplarily, cosine similarity may be used to calculate the distance between the text description vector of the query graph and the text description vector of the second candidate product, thereby obtaining the text-to-text similarity between the query graph and the second candidate product.

[0043] Exemplarily, a weighted sum is performed on the text-to-image similarity and the text-to-text similarity between the query graph and the second candidate product to obtain a second comprehensive score for the query graph and the second candidate product.

[0044] According to the above implementation, based on the graph vector of the query graph and the graph vector of each perspective graph of each first candidate commodity, the graph-to-graph similarity between the query graph and each first candidate commodity is determined; based on the graph vector of the query graph and the text description vector of each first candidate commodity, the graph-to-text similarity between the query graph and each first candidate commodity is determined; based on the graph-to-graph similarity and the graph-to-text similarity between the query graph and each first candidate commodity, the first comprehensive score between the query graph and each first candidate commodity is determined. In this way, the input query graph can be used to evaluate the graph-to-graph similarity and the graph-to-text similarity between the query graph and the candidate commodities, thereby achieving the effect of graph retrieval. In the case where the first comprehensive score of the query graph and each first candidate product does not meet the preset scoring conditions, the query graph is semantically parsed to obtain the text description vector of the query graph; based on the text description vector of the query graph and the graph vector of each perspective graph of each second candidate product, the text-to-image similarity between the query graph and each second candidate product is determined; based on the text description vector of the query graph and the text description vector of each second candidate product, the text-to-text similarity between the query graph and each second candidate product is determined; based on the text-to-image similarity and text-to-text similarity between the query graph and each second candidate product, the second comprehensive score of the query graph and each second candidate product is determined; based on the second comprehensive score of the query graph and each second candidate product, the target candidate product is determined among the second candidate products. In this way, when the graph retrieval is not ideal, the query graph can be semantically parsed to obtain the text description information of the query graph, so that the text description information of the query graph can be used for graph-text retrieval to obtain the corresponding retrieval results. Therefore, the technical solution of the present invention can improve the accuracy of graph retrieval.

[0045] In one embodiment, the above method may further include: When the query graph and the first comprehensive scores of the first candidate products meet the scoring conditions, the first candidate products are sorted based on the query graph and the first comprehensive scores of the first candidate products to obtain product search results of the query graph.

[0046] It can be understood that the query graph and the first comprehensive scores of the first candidate products meet the scoring conditions, which means that the results obtained by performing graph search on the query graph are relatively ideal.

[0047] According to the above implementation, when the results of graph search for the query graph are ideal, the first candidate products can be directly sorted based on the query graph and the first comprehensive score of each first candidate product to obtain the product search results of the query graph, that is, the feedback graph search results.

[0048] In one embodiment, the method further includes: upon receiving a query graph, determining a first candidate product cluster in each candidate product cluster based on the distance between a graph vector of the query graph and a cluster center vector of each candidate product cluster; and determining each first candidate product based on the first candidate product cluster.

[0049] In one embodiment, the method further includes: determining a second candidate commodity cluster in each candidate commodity cluster based on the distance between the text description vector of the query graph and the cluster center vector of each candidate commodity cluster; and determining each second candidate commodity based on the second candidate commodity cluster.

[0050] Exemplarily, KL divergence or cosine similarity is used to calculate the distance between the graph vector of the query graph and the cluster center vector of each candidate product cluster.

[0051] Exemplarily, KL divergence or cosine similarity is used to calculate the distance between the text description vector of the query graph and the cluster center vector of each candidate product cluster.

[0052] For example, when the system is initialized or the vector database is updated, clustering operations are performed on the product image vectors and product name description vectors in the vector database. These vectors are divided into N clusters by the k-means clustering algorithm, and the cluster center of each cluster is determined. N is a positive integer.

[0053] According to the above implementation, when a query graph vector is received, its similarity with each cluster center is first calculated, and the top-N clusters with the closest distance are selected. For the selected N clusters, the product graph vector and product name description vector in the cluster are obtained. The similarity between the query graph vector and the vector in the cluster is calculated, and the above-mentioned conversion to residual and other operations are used to improve the calculation efficiency and accuracy.

[0054] In one embodiment, based on the graph vector of the query graph and the graph vector of each perspective graph of each first candidate product, the graph-to-graph similarity between the query graph and each first candidate product is determined, including: based on the graph vector of the query graph and the graph vector of each perspective graph of the first candidate product, determining the similarity between the query graph and each perspective graph of the first candidate product; based on the maximum value of the similarities between the query graph and each perspective graph of the first candidate product, determining the graph-to-graph similarity between the query graph and the first candidate product.

[0055] In one embodiment, based on the text description vector of the query graph and the graph vector of each perspective graph of each second candidate product, the text and image similarity between the query graph and each second candidate product is determined, including: based on the text description vector of the query graph and the graph vector of each perspective graph of the second candidate product, determining the text and image similarity between the query graph and each perspective graph of the second candidate product; based on the maximum value of the text and image similarities between the query graph and each perspective graph of the second candidate product, determining the text and image similarity between the query graph and the second candidate product.

[0056] According to the above implementation, since there are multiple view graphs of the candidate product, there are multiple similarities between the graph vector of the query graph and the graph vectors of each view graph, from which the maximum value is selected as the graph-to-graph similarity between the query graph and the candidate product. Similarly, there are multiple similarities between the text description vector of the query graph and the graph vectors of each view graph, from which the maximum value is selected as the text-to-graph similarity between the query graph and the candidate product. In this way, the graph-to-graph similarity and the text-to-graph similarity between the query graph and the candidate product can be accurately evaluated.

[0057] In one embodiment, semantic parsing is performed on a query graph to obtain a text description vector of the query graph, including: obtaining prompt words of the query graph; inputting the query graph and the prompt words of the query graph into a graph-text understanding model to obtain a text description vector of the query graph output by the graph-text understanding model.

[0058] Exemplarily, the image-text understanding model may be a pre-trained large language model.

[0059] It is understandable that a semantic decoupling retrieval module can be used to perform the above semantic analysis and the subsequent retrieval process for the second candidate product. When the comprehensive similarity score of the image and text vector, i.e., the above first comprehensive score, does not meet the preset scoring conditions, the semantic decoupling retrieval module is triggered to perform the above steps. The main function of this module is to further process the query graph that exceeds the conventional retrieval range to obtain retrieval results that better meet user needs.

[0060] For example, call an external image-text understanding model and use the pre-set prompt words to perform semantic analysis on the query graph. In this process, the large model will conduct a comprehensive analysis based on the visual information in the query graph and the small amount of text information that may exist. For example, if the query graph is a product image with some brand logos and product features, the model will identify these key information and further organize and interpret these information according to the guidance of the prompt words. Based on the extraction of the product appearance and some text, output its "brand name-product name-specification / color" and other description information. This description information is the key basis for subsequent retrieval of similar products. At the same time, in order to be able to use CLIP's cross-modal capabilities for retrieval, it is necessary to convert these text description information into vector form. Usually, CLIP is used to encode the product name text (the product name here is a combination of the above-mentioned output description information) to generate a product name description vector.

[0061] According to the above implementation, the query graph is semantically parsed to obtain the corresponding text description vector. Therefore, when the graph retrieval is not ideal, the text description vector is used to perform text-graph and text-text retrieval to obtain the final retrieval result, thereby improving the accuracy of the retrieval.

[0062] In some examples, the interface service module can also interact with external systems to provide a service interface for the image recognition and product search system. The interface service module receives requests from the backend of the mall platform and returns the search results to the mall platform. The interface service module receives requests from the backend of the mall platform through the HTTP protocol. The request contains parameters such as the URL of the query graph and the top-k value. When a request is received, the interface service module will first parse the request and extract relevant parameter information. The received request is verified to check whether the format of the request is correct and whether the parameters are complete and meet the requirements. If there is a problem with the request, such as the lack of necessary parameters or the wrong parameter format, the corresponding error message will be returned to the backend of the mall platform. In order to cope with high concurrency requests, the interface service module adopts a request queuing mechanism. When multiple requests are received, they are placed in the request queue in the order in which they are received.

[0063] At the same time, in order to make full use of system resources and improve processing efficiency, the number of requests processed simultaneously will be reasonably controlled according to the concurrent processing capability of the system. For example, if the system can process n requests simultaneously, then when the number of requests in the request queue reaches n, subsequent requests will wait in the queue until a processed request vacates a place. Once the request is verified and enters the processing flow, the interface service module obtains the image data from the URL of the query graph. For the obtained images, they are not processed individually, but the image batch processing calculation method based on the large model is adopted. The image data in multiple requests are grouped to form image batches. For example, according to the memory and computing resources of the system, each batch can contain m images. Then these image batches are sent to the image-encoder of the Chinese-CLIP model for encoding processing in turn. This batch processing method can make full use of the parallel computing power of the large model and improve the efficiency of image encoding. After the image vectorization and retrieval module call, the result list is obtained and returned through json.

[0064] The embodiments of the present invention have the following beneficial effects: 1. Utilize the cross-modal comparison capability of the Chinese-CLIP model to establish a comprehensive scoring ranking of product images and product name descriptions, taking into account both implicit feature expression and explicit semantic expression, and improving the accuracy of retrieving identical or similar products.

[0065] 2. When establishing a product image vector database, a one-to-many mapping is performed between the product ID and the vector to store the product. The similarity between the query image and the multiple view image vectors of each product is calculated, and the maximum similarity score is selected for subsequent scoring. This makes full use of the multi-view images of the product to improve the comprehensiveness and accuracy of the search results.

[0066] 3. When the comprehensive similarity score of the image and text vector does not meet the scoring conditions, the external image and text understanding model is used for semantic analysis, and the text is vectorized based on the analysis results to retrieve similar products, solving the problem of unsatisfactory retrieval results when users enter special product images.

[0067] 4. By performing multi-threading and asynchronous optimization on query requests and batch processing optimization on images, the efficiency of inference model calls can be improved, high-concurrency query requirements can be met, and user waiting time can be reduced.

[0068] Figure 4 It is a structural block diagram of a commodity search device according to an embodiment of the present invention.

[0069] like Figure 4 As shown, the commodity search device may include: A first similarity calculation module 410, configured to determine the image-image similarity between the query image and each of the first candidate products based on the image vector of the query image and the image vector of each of the perspective images of each of the first candidate products; A second similarity calculation module 420, configured to determine the image and text similarity between the query graph and each of the first candidate products based on the image vector of the query graph and the text description vector of each of the first candidate products; A first scoring module 430, configured to determine a first comprehensive score of the query graph and each of the first candidate products based on the image-image similarity and image-text similarity between the query graph and each of the first candidate products; A semantic parsing module 440, configured to perform semantic parsing on the query graph to obtain a text description vector of the query graph when the query graph and the first comprehensive scores of the first candidate products do not meet a preset score condition; A third similarity calculation module 450, configured to determine the text and image similarity between the query graph and each of the second candidate products based on the text description vector of the query graph and the image vector of each of the perspective images of each of the second candidate products; A fourth similarity calculation module 460, configured to determine the text similarity between the query graph and each of the second candidate commodities based on the text description vector of the query graph and the text description vector of each of the second candidate commodities; A second scoring module 470, configured to determine a second comprehensive score of the query graph and each of the second candidate commodities based on the text-to-picture similarity and text-to-text similarity between the query graph and each of the second candidate commodities; The first result determination module 480 is configured to sort the second candidate commodities based on the query graph and the second comprehensive scores of the second candidate commodities to obtain commodity search results of the query graph.

[0070] In one embodiment, the above device further comprises: A second result determination module is used to sort each of the first candidate products based on the query graph and the first comprehensive score of each of the first candidate products to obtain product search results of the query graph when the query graph and the first comprehensive score of each of the first candidate products meet the scoring conditions.

[0071] In one embodiment, the above device further comprises: A first product cluster determination module, configured to determine a first candidate product cluster from among the candidate product clusters based on the distance between the graph vector of the query graph and the cluster center vector of each candidate product cluster when the query graph is received; The first candidate commodity determining module is configured to determine each of the first candidate commodities based on the first candidate commodity cluster.

[0072] In one embodiment, the above device further comprises: A second product cluster determination module, configured to determine a second candidate product cluster in each of the candidate product clusters based on the distance between the text description vector of the query graph and the cluster center vector of each of the candidate product clusters; The second candidate commodity determining module is used to determine each of the second candidate commodities based on the second candidate commodity cluster.

[0073] In one implementation, the first similarity calculation module 410 includes: a first calculation unit, configured to determine the similarity between the query graph and each of the perspective graphs of the first candidate product based on the graph vector of the query graph and the graph vectors of each of the perspective graphs of the first candidate product; The second calculation unit is used to determine the image-to-image similarity between the query image and the first candidate product based on the maximum value of the similarities between the query image and each view image of the first candidate product.

[0074] In one implementation, the third similarity calculation module 450 includes: a third calculation unit, configured to determine text and image similarities between the query graph and each of the perspective images of the second candidate product based on the text description vector of the query graph and the image vectors of each of the perspective images of the second candidate product; The fourth calculation unit is used to determine the text-image similarity between the query graph and the second candidate product based on the maximum value of the text-image similarities between the query graph and each perspective graph of the second candidate product.

[0075] In one implementation, the semantic parsing module 440 includes: A prompt word acquisition module, used to acquire the prompt word of the query graph; The model processing module is used to input the query graph and the prompt words of the query graph into the image-text understanding model to obtain the text description vector of the query graph output by the image-text understanding model.

[0076] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present invention, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0077] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0078] According to an embodiment of the present invention, the present invention also provides a system and a readable storage medium.

[0079] Figure 5 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0080] like Figure 5 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0081] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0082] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as access control policy mining and access control methods. For example, in some embodiments, the access control policy mining and access control methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the access control policy mining and access control methods described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the access control policy mining and access control method in any other appropriate manner (eg, by means of firmware).

[0083] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0084] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0085] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0086] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0087] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0088] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0089] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0090] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A commodity search method based on multimodality, characterized in that: include: Based on the graph vector of the query graph and the graph vectors of each perspective graph of each first candidate commodity, determining the graph-to-graph similarity between the query graph and each of the first candidate commodities; Based on the graph vector of the query graph and the text description vector of each of the first candidate products, determining the graph and text similarity between the query graph and each of the first candidate products; Determine a first comprehensive score for the query graph and each of the first candidate products based on the image-image similarity and image-text similarity between the query graph and each of the first candidate products; When the query graph and the first comprehensive scores of the first candidate products do not meet a preset score condition, semantically analyze the query graph to obtain a text description vector of the query graph; Determine the text and image similarity between the query graph and each of the second candidate products based on the text description vector of the query graph and the image vector of each of the perspective images of each of the second candidate products; Determining text similarity between the query graph and each of the second candidate commodities based on the text description vector of the query graph and the text description vector of each of the second candidate commodities; Determine a second comprehensive score for the query graph and each of the second candidate products based on the text-to-image similarity and text-to-text similarity between the query graph and each of the second candidate products; Based on the query graph and the second comprehensive scores of the second candidate commodities, the second candidate commodities are sorted to obtain commodity search results of the query graph.

2. The method according to claim 1, characterized in that Also includes: When the query graph and the first comprehensive scores of the first candidate products meet the scoring conditions, the first candidate products are sorted based on the query graph and the first comprehensive scores of the first candidate products to obtain product search results of the query graph.

3. The method according to claim 1, characterized in that Also includes: When the query graph is received, determining a first candidate product cluster from among the candidate product clusters based on the distance between the graph vector of the query graph and the cluster center vector of each candidate product cluster; Based on the first candidate commodity cluster, each of the first candidate commodities is determined.

4. The method according to claim 3, characterized in that Also includes: Determine a second candidate product cluster in each of the candidate product clusters based on the distance between the text description vector of the query graph and the cluster center vector of each of the candidate product clusters; Based on the second candidate commodity cluster, each of the second candidate commodities is determined.

5. The method according to claim 1, characterized in that The determining, based on the graph vector of the query graph and the graph vectors of each view graph of each first candidate commodity, the graph-to-graph similarity between the query graph and each first candidate commodity comprises: Determining the similarity between the query graph and each of the perspective graphs of the first candidate product based on the graph vector of the query graph and the graph vectors of each of the perspective graphs of the first candidate product; Based on the maximum value of the similarities between the query graph and each of the perspective graphs of the first candidate product, the image-to-image similarity between the query graph and the first candidate product is determined.

6. The method according to claim 1, characterized in that The determining, based on the text description vector of the query graph and the image vector of each view image of each second candidate product, the text and image similarity between the query graph and each second candidate product includes: Determine, based on the text description vector of the query graph and the image vectors of each perspective image of the second candidate product, the text and image similarity between the query graph and each perspective image of the second candidate product; Based on the maximum value of the text-image similarities between the query graph and each view graph of the second candidate product, the text-image similarity between the query graph and the second candidate product is determined.

7. The method according to any one of claims 1 to 6, characterized in that The performing semantic parsing on the query graph to obtain a text description vector of the query graph includes: Obtaining a prompt word of the query graph; The query graph and the prompt words of the query graph are input into a graph-text understanding model to obtain a text description vector of the query graph output by the graph-text understanding model.

8. A multi-modal commodity search device, characterized in that: include: A first similarity calculation module, configured to determine the image-image similarity between the query image and each of the first candidate products based on the image vector of the query image and the image vector of each of the perspective images of each of the first candidate products; a second similarity calculation module, configured to determine the image and text similarity between the query graph and each of the first candidate products based on the graph vector of the query graph and the text description vector of each of the first candidate products; A first scoring module, configured to determine a first comprehensive score of the query graph and each of the first candidate commodities based on a graph-graph similarity and a graph-text similarity between the query graph and each of the first candidate commodities; A semantic parsing module, configured to perform semantic parsing on the query graph to obtain a text description vector of the query graph when the query graph and the first comprehensive scores of each of the first candidate products do not meet a preset score condition; A third similarity calculation module, configured to determine the text and image similarity between the query graph and each of the second candidate products based on the text description vector of the query graph and the image vector of each of the perspective images of each of the second candidate products; a fourth similarity calculation module, configured to determine the text similarity between the query graph and each of the second candidate commodities based on the text description vector of the query graph and the text description vector of each of the second candidate commodities; A second scoring module, configured to determine a second comprehensive score of the query graph and each of the second candidate commodities based on a text-to-picture similarity and a text-to-text similarity between the query graph and each of the second candidate commodities; The first result determination module is used to sort each of the second candidate commodities based on the query graph and the second comprehensive score of each of the second candidate commodities to obtain a commodity search result of the query graph.

9. A commodity search system, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images