Visual search method and device
Patent Information
- Application Number
- CN202280095518.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-25
- Publication Date
- 2024-12-06
AI Technical Summary
Existing visual search technology has poor flexibility and cannot help users fully describe search intentions, resulting in low accuracy of search results and poor user experience.
Obtain users' search intentions through multiple rounds of interactions, use multi-level query recommendation libraries and multi-modal information fusion to help users describe and improve search intentions efficiently and clearly, actively explore users' potential points of interest, and improve the effectiveness and flexibility of search sex.
It achieves a more efficient and flexible visual search, can accurately identify users' complex search intentions, and improves the accuracy of search results and user experience.
Smart Images

Figure CN119096239A_ABST
Abstract
Description
Visual search method and device Technical Field
[0001] The present application relates to the field of search technology, and in particular to a visual search method and device. Background Art
[0002] Visual search is a key technology in the internet sector, with typical applications such as "image search" and "image text search." Through visual search, a professional search engine system provides users with retrieval services for relevant graphic and image data on the internet. This is a subdivision of search engines, such as Microsoft's "Bing" search engine, which uses images to help users more conveniently complete specific search tasks. In the current digital age where consumer attention spans and time are sharply declining, effectively using visual search to capture users' real needs and enhance their consumption experience is increasingly becoming a consensus among major e-commerce platforms. Furthermore, a Data Bridge survey indicates that the market valuation of visual search will grow from US$6 billion to US$30 billion. This rapidly growing market continues to drive the iterative development of related visual search technologies.
[0003] However, the visual search in the existing technology has poor flexibility and can only passively respond to user queries. It cannot help users identify or improve unclear search intentions, which leads to low search result accuracy and poor user experience.
[0004] Summary of the Invention
[0005] The embodiments of the present application provide a visual search method and device, which, through multiple rounds of interaction, help users describe their search intentions efficiently, clearly, and completely, guide and improve their search intentions, actively explore their potential points of interest, and improve the effectiveness and flexibility of the search.
[0006] In a first aspect, the present application provides a visual search method, comprising: obtaining an image to be searched; obtaining a first round of search results based on features of the image to be searched and features of first-level objects in a query recommendation library, the first round of search results including multiple qualified first-level objects; wherein the query recommendation library includes N levels of objects, each N-1th level object corresponds to multiple Nth level objects, N is an integer greater than 1, and the objects include text content and / or image content and / or video content and / or audio content; late interactively fusing the features of the image to be searched and the features of the first target object to obtain a first cumulative search intent feature, the first target object being an object selected by the user from multiple qualified first-level objects; obtaining a second round of search results based on the first cumulative search intent feature, the second round of search results including multiple qualified second-level objects corresponding to the first target object.
[0007] In this possible implementation, through multiple rounds of interaction, users can be helped to describe their search intentions efficiently, clearly, and completely, guide and improve their search intentions, actively explore their potential interests, and improve the effectiveness and flexibility of the search.
[0008] In a possible implementation, a first-level object whose similarity to the image to be searched is greater than a preset threshold is determined as a first-level object that meets the criteria.
[0009] In another possible implementation, multiple qualified first-level objects in the first round of search results are sorted from high to low according to their similarity to the image to be searched.
[0010] In another possible implementation, a second-level object whose similarity with the first cumulative search intent feature is greater than a preset threshold is determined as a qualified second-level object.
[0011] In another possible implementation, multiple qualified second-level objects in the second round of search results are sorted from high to low according to their similarity to the first cumulative search intent feature.
[0012] In another possible implementation, the Mth cumulative intent feature is late interactively fused with the feature of the Lth target object to obtain the final search intent, wherein the Lth target object is the object selected by the user from multiple qualified Lth-level objects, M is a positive integer greater than or equal to 1, and L is a positive integer greater than M; based on the final search intent, the final search result is obtained, and the final search result includes the qualified L+1th-level object corresponding to the Lth target object.
[0013] In one example, the final search intent is further related to the first text feature, where the first text feature is a feature of the query text input by the user.
[0014] In another possible implementation, the final search results include card search results and / or expanded search results.
[0015] In another possible implementation, the query recommendation library includes information in multiple modalities, and the information in multiple modalities is a tree structure. The nodes of the tree structure represent the objects, and the nodes at different levels of the tree structure represent objects at different levels.
[0016] In a second aspect, the present application provides a visual search device, comprising:
[0017] An acquisition module, used to acquire the image to be searched;
[0018] a cumulative search intent determination module, configured to obtain a first round of search results based on features of the image to be searched and features of first-level objects in the query recommendation library, wherein the first round of search results includes a plurality of first-level objects that meet the criteria;
[0019] The query recommendation library includes N levels of objects, each of the N-1th level objects corresponds to multiple Nth level objects, N is an integer greater than 1, and the objects include text content and / or image content and / or video content and / or audio content;
[0020] Performing late interactive fusion on the features of the image to be searched and the features of the first target object to obtain a first cumulative search intent feature, where the first target object is an object selected by the user from the plurality of first-level objects that meet the criteria;
[0021] The search result determination module is used to obtain a second round of search results based on the first cumulative search intention feature, where the second round of search results includes multiple qualified second-level objects corresponding to the first target object.
[0022] In a possible implementation, a first-level object whose similarity with the image to be searched is greater than a preset threshold is determined as a first-level object that meets the criteria.
[0023] In another possible implementation, the plurality of first-level objects that meet the criteria in the first round of search results are sorted from high to low according to their similarity to the image to be searched.
[0024] In another possible implementation, a second-level object whose similarity with the first cumulative search intent feature is greater than a preset threshold is determined as a qualified second-level object.
[0025] In another possible implementation, multiple qualified second-level objects in the second round of search results are sorted from high to low according to their similarity to the first cumulative search intent feature.
[0026] In another possible implementation, the search result determination module is further configured to perform late interactive fusion of the Mth accumulated intent feature with the feature of the Lth target object to obtain a final search intent, wherein the Lth target object is an object selected by the user from the plurality of Lth-level objects that meet the criteria, M is a positive integer greater than or equal to 1, and L is a positive integer greater than M;
[0027] Based on the final search intent, a final search result is obtained, where the final search result includes an L+1th level object that meets the criteria and corresponds to the Lth target object.
[0028] In another possible implementation, the final search intent is also related to a first text feature, where the first text feature is a feature of the query text input by the user.
[0029] In another possible implementation, the final search results include card search results and / or expanded search results.
[0030] In another possible implementation, the query recommendation library includes information in multiple modalities, and the information in multiple modalities is a tree structure. The nodes of the tree structure represent the objects, and the nodes at different levels of the tree structure represent objects at different levels.
[0031] In a third aspect, the present application provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described in the first aspect.
[0032] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect of the present application.
[0033] In a fifth aspect, the present application provides a computer program or a computer program product, wherein the computer program or computer program product includes instructions, and when the instructions are executed, the method described in the first aspect of the present application is implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] FIG1 shows a schematic diagram of a visual search process;
[0035] FIG2 is an architecture diagram of a visual search system provided in an embodiment of the present application;
[0036] FIG3 is a schematic diagram of the construction process of the query recommendation library;
[0037] FIG4 is a schematic diagram of query recommendation in a query recommendation library during a search process;
[0038] Figure 5a is a schematic diagram of card search results;
[0039] Figure 5b is a schematic diagram of expanding search results;
[0040] FIG6 is a flowchart of a visual search method provided in an embodiment of the present application;
[0041] FIG7 is a schematic diagram of the structure of a visual search device provided in an embodiment of the present application;
[0042] FIG8 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] The technical solution of the present application is further described in detail below through the accompanying drawings and examples.
[0044] In order to better understand the technical solutions provided by the embodiments of the present application, some of the terms are briefly introduced below.
[0045] query: The content entered by the user in the search box.
[0046] Semantic space: the world of linguistic meaning. Each symbol system is, in a broad sense, a language that conveys meaning, and the meanings they express constitute a specific semantic space.
[0047] Semantic features: The basic concepts and meanings of the content are represented by feature numerical vectors.
[0048] Modality: Every source or form of information can be called a modality.
[0049] Cross-modal retrieval: Information retrieval often requires more than just data from a single modality for the same event. Data from other modalities may also be needed to enrich our understanding of the same thing or event. In this case, cross-modal retrieval is needed to enable retrieval between data from different modalities.
[0050] Multi-source fusion: Integrate various data information, absorb the characteristics of different data sources, and then extract unified information from them, which is better and richer than single data.
[0051] Vector retrieval: In a given vector dataset, retrieve K vectors that are close to the query vector according to a certain metric.
[0052] Graph: A structure that represents the connections between some things, objects, and entities and other things, objects, and entities.
[0053] Figure 1 shows a schematic diagram of a visual search process. As shown in Figure 1, in order to complete the technical implementation of visual search, visual search includes the following steps:
[0054] 1) Offline database construction. Using data mining technology, key information is obtained and filtered from various structured (such as product databases) or unstructured (such as web pages) data sources to achieve offline database construction operations.
[0055] 2) Online, we use BERT to calculate semantic features for text and SWIN TRANSFORMER to calculate semantic features for images based on the user's actual query content. The resulting features are then projected into the same semantic space as the offline corpus.
[0056] 3) Based on the actual query content features, matching retrieval and rough recall based on feature similarity are performed from the offline database;
[0057] 4) Further combine the rough recall recommendation results for fine ranking and return the final recommendation results.
[0058] However, faced with the effectiveness of multimodal information and recommendations in actual scenarios, the recommendations of this visual search method are becoming increasingly insufficient.
[0059] In order to solve the above problems, the embodiments of the present application propose several visual search solutions.
[0060] The first visual search solution includes: Content database construction: After cleaning product images and web page images, an image database is constructed. Using data mining technology, labels are extracted from various structured (such as product databases) or unstructured (such as web pages) data sources. After deduplication and cleaning, a text label database is constructed.
[0061] Content semantic feature calculation: Using the text tower and image tower of the multimodal semantic model, we calculate the content feature vectors of each tag and image in the database.
[0062] Query semantic feature calculation: Calculate the semantic feature vector of the user's query image through the image tower of the multimodal semantic model;
[0063] Content search (image search, image search for text): Calculate the similarity between query features and the features of text and images in the content base, and return search content with higher similarity to the user, ultimately achieving the effect of visual search.
[0064] The first visual search solution, based on multimodal semantic matching, calculates image and text modal features, and can, to a certain extent, accurately determine the general relevance between query images and database content, thereby achieving the goal of visual search. However, it still has shortcomings such as poor search flexibility and passive response.
[0065] Poor search flexibility: Only single-pass image queries are supported, preventing users from fully describing complex search intent, such as "How can I repair this situation (in the image)?" Given limited input intent, the system can only complete simple search requests like finding similarities and identifying objects by calculating general semantic relevance.
[0066] Passive response: It can only respond to user queries passively, and cannot help users identify and improve unclear search intentions. It also cannot actively stimulate user search interests, thus restricting search traffic and duration.
[0067] The second visual search solution is to populate and expand based on the user's search content, providing information that the search engine deems relevant to the topic, thereby helping users obtain recommended information faster and better. This solution is used to handle complex multimodal searches. The visual search experience innovation is further explained. From a technical perspective, it allows users to enter additional text queries on top of basic visual search, initiating a fusion search of images and text. This significantly improves search flexibility and can support some search intents that traditional search technologies cannot fulfill.
[0068] The second visual search solution has the following problems:
[0069] Passive response: It can only respond to user queries passively and cannot help users identify unclear search intentions. Similarly, this solution cannot actively stimulate user search interest and restricts search traffic and duration.
[0070] Complex operation: On the one hand, users need to input queries twice (upload pictures + enter text) to complete complex intent searches, which goes against the user mentality of visual search, which is "one-click, get results"; on the other hand, if users are not satisfied with the search results, they need to re-enter the query every time.
[0071] The third visual search solution includes:
[0072] (1) Using the representation model to calculate the feature vector representation of the user input content;
[0073] (2) Perform similarity search in the offline constructed database based on the feature vector of the input content and return the sorted results;
[0074] (3) Based on the user input and the returned results, provide the user with further refined search recommendations;
[0075] (4) After the user interacts with the refined search recommendation items, the feature vector of the content is calculated again and used as an anchor point to perform a secondary search in the recommended results returned in the previous round;
[0076] (5) Repeat the above dialogue interaction process until the user is guided to complete the effective recommendation of the search intention.
[0077] This technical solution uses user input (text, images) to represent the input in a vector space. By integrating text and images, the search scope is continuously refined, enabling interactive and complex queries. This solution improves the ability to handle complex queries through a dialogue-based approach.
[0078] The third visual search method, based on multimodal conversational search, integrates image and text, adds multi-round interaction capabilities to improve the efficiency of complex queries, but still has the following shortcomings:
[0079] (1) Passive response: It can only respond to user queries passively and cannot help users identify unclear search intent;
[0080] (2) Complex operation: users are required to continuously refine their query requirements, which seriously affects user input content and reduces user experience;
[0081] (3) High technical difficulty: The relevant supporting technologies mainly exist in the academic circle and are far from being implemented in the industry, making them difficult to commercialize.
[0082] In order to solve the problems existing in the above-mentioned solutions and existing technologies, the embodiment of the present application proposes a visual search method, which is based on a multi-level query recommendation library, fully mines structured and unstructured multi-source data, automatically constructs a multi-level tree-like query recommendation base library, and uses a late-interactive multi-source content fusion strategy. Combined with user behavior (inputting pictures, clicking, etc.), it continuously updates query recommendations from two dimensions of breadth and depth, helping users to describe their search intentions efficiently, clearly and completely, guide and improve users' search intentions, actively mine users' potential points of interest, and improve the effectiveness and flexibility of search.
[0083] Figure 2 is an architecture diagram of a visual search system provided in an embodiment of the present application. The system mainly includes two module components: offline and online.
[0084] The offline module components include: query recommendation library construction module, multimodal content library, multi-level query recommendation library, and content base library construction module.
[0085] The online module components include: human-computer interaction module, multimodal information understanding module, multi-information fusion module, and semantic vector retrieval module.
[0086] The query recommendation library construction module in the offline module component is used to construct tree stumps using structured data such as graphs and multi-level tags; it mines high-frequency words from web pages / logs to expand the number of root nodes. It expands relationship nodes in terms of depth and breadth, and uses tools such as synonyms and language models to deduplicate nodes, merging subtrees or leaf nodes mounted under duplicate nodes.
[0087] The multimodal information understanding module in the online module component is used to make recommendations based on query content features and query recommendation similarity calculations. It also improves user query intent based on user online behavior accumulation and is used for adjusting user intent recommendations.
[0088] The multi-information fusion module in the online module: performs late interaction fusion modeling based on the accumulated intent features and the next-layer node query recommendation text features, retrieves the various modal information in the content base, and returns the Top-1 as the query recommendation details of the node; further expands the search, performs extended search based on the accumulated intent features (and can further integrate the user's additional input text features), and returns more content information.
[0089] This visual search system is a multi-level conversational query recommendation system. Its main function is to interact with the accumulated user behavior based on the query image entered by the user and the interactive click-and-swipe recommendation results, guide and improve the user's search intent, actively explore the user's potential points of interest, improve the effectiveness and flexibility of the search, and return effective retrieval results to the user. The main task of the offline module component is to extract structural information such as the upper and lower levels of query recommendations from multi-source data and construct a multi-level (tree-like) query recommendation library. The main task of the online module component is to guide and improve the user's search intent through conversational cross-modal query recommendations and multi-faceted fusion content, and continue the interaction until the search is completed.
[0090] The implementation steps are as follows:
[0091] S1. Deploy the system on a specific hardware server. The offline module components and the online module components can be deployed on the same hardware server or separately.
[0092] S2, in the offline stage, extracts structural information such as the upper and lower levels of query recommendations from multi-source data, builds a multi-level (tree-like) query recommendation base, and completes the offline construction of the multi-level query recommendation base. Through structured computing logic such as multi-level category labels of the data, the semantic similarity is expanded in depth (based on the inclusion relationship with synonymous judgment to expand child nodes) and breadth (based on the intersection relationship with synonymous judgment and the co-occurrence relationship in the same picture to expand sibling nodes, and then add them after verifying the relationship with the parent node). At the same time, based on tools such as synonym dictionaries and language models, nodes are deduplicated and the subtrees or leaf nodes mounted under the duplicate nodes are merged. The specific implementation is shown in Figure 3.
[0093] S3, the online recommendation stage, guides user interaction based on the query image and the multi-level query recommendation library, and continuously improves the accumulated intent features until the search is completed (supporting rollback during the process). First, for the first-level nodes (i.e., the branch nodes closest to the root node), based on the similarity between the query image features and the query recommendation features, the first-level recommendations are selected and sorted from all nodes; for the subsequent layers, the child nodes of the user-selected node are returned as candidates for the next layer; then, based on the query image features and the query recommendation text features selected by the user, late interaction fusion modeling is performed to improve the accumulated intent features. Finally, based on the similarity between the current accumulated intent features and the query recommendation text features of the child nodes, the next-level nodes are pruned and re-ranked online (see Figure 4).
[0094] S4. Based on the accumulated intent features, the next layer query recommendation features, and the user's additional input text features, a multi-faceted fusion is performed to retrieve various modal information in the content base, and finally provide card search (see Figure 5a) and / or expanded search results (see Figure 5b).
[0095] FIG6 is a flowchart of a visual search method provided in an embodiment of the present application. The visual search method can be implemented by the visual search system shown in FIG2. As shown in FIG6, the visual search method provided in an embodiment of the present application includes steps S601 to S604.
[0096] In step S601, an image to be searched is obtained.
[0097] The image to be searched can be received by the search terminal as user input and uploaded to the server. The search terminal (e.g., a smartphone) can obtain the image to be searched by capturing it with a camera (e.g., a mobile phone camera), or it can directly retrieve an image from local storage as the image to be searched. This application does not limit the specific method of obtaining the image to be searched.
[0098] In step S602, based on the features of the image to be searched and the features of the first-level objects in the query recommendation library, a first-round search result is obtained, and the first-round search result includes a plurality of first-level objects that meet the criteria.
[0099] Extract the semantic features of the image to be searched, for example, by extracting the semantic features of the image to be searched through the SWIN TRANSFORMER model, map the semantic features of the image to be searched into the same semantic space as the query recommendation library, and obtain a semantic feature vector of the image to be searched; then calculate the similarity between the semantic feature vector of the image to be searched and the features of the first-level objects in the query recommendation library, determine the first-level objects with a similarity greater than a preset threshold (for example, 0.8) as qualified objects, and use multiple qualified first-level objects as the search results of the first round.
[0100] Among them, the query recommendation library includes N levels of objects, each N-1th level object corresponds to multiple Nth level objects, N is an integer greater than 1, and the objects include text content and / or image content and / or video content and / or audio content; that is, the information in the query recommendation library is multimodal information, including text content information, image content information, audio content information, etc.
[0101] Optionally, the multi-modal information in the query recommendation library is structured in a tree, where nodes in the tree represent objects, and nodes at different levels of the tree represent objects at different levels. The construction of the query recommendation library and the specific structure of the tree structure can be found in the description of the query recommendation library above and will not be repeated here for the sake of brevity.
[0102] By setting up a query recommendation library, we can fully mine structured and unstructured multi-source data, automatically construct a multi-level tree-like query recommendation library, and structure the potential paths of user information retrieval and information exploration, providing effective, low-cost and scalable data support for the implementation of the visual search method provided in the embodiment of this application.
[0103] In one example, multiple first-level objects that meet the criteria in the first round of search results are sorted from high to low based on their similarity to the image being searched. For example, as shown in Figure 4, after calculating the similarity between the image being searched and each first-level object, the first-level objects that meet the similarity criteria are sorted from high to low based on their similarity: bicycle parts, screws, and wire. In other words, the first round of search results will be sorted by their similarity to the image being searched and displayed to the user. The higher the similarity, the closer it is to the user's initial search intent, and the more likely it is to be the content the user is searching for. The higher the ranking, the easier it is for the user to find the content they are searching for.
[0104] In step S603, the features of the image to be searched and the features of the first target object are late interactively fused to obtain a first cumulative search intention feature, where the first target object is the object selected by the user from multiple first-level objects that meet the criteria.
[0105] After obtaining the feature vector of the image to be searched and the feature vector of the first target object, the feature vector of the image to be searched and the feature vector of the first target object are weightedly fused to obtain a first cumulative search intent. The weights of the feature vector of the image to be searched and the feature vector of the first target object can be determined in various ways, such as by system default or user setting.
[0106] The first target object is a first-level object selected by the user. For example, the user clicks on a first-level object in the first round of search results on the screen as the first target object, such as the bicycle accessories in FIG. 4 .
[0107] The user's initial search intention, which may not be complete, can be obtained from the features of the image to be searched. The features of the first target object selected by the user (which may be text features) reflect the user's further search intention. By combining the features of the image to be searched (and the initial search intention) and the features of the first target object (and the further search intention), a more complete cumulative search intention can be obtained.
[0108] In step S604, based on the first accumulated search intent feature, a second round of search results is obtained, where the second round of search results includes a plurality of qualified second-level objects corresponding to the first target object.
[0109] Calculate the similarity between the first cumulative intent feature and each second-level object feature corresponding to the first target object. Objects with a similarity greater than a preset threshold (for example, 0.8) are determined to be qualified second-level objects, and the qualified second-level objects are used as search results for the second round.
[0110] Optionally, multiple qualified second-level objects in the second round of search results are sorted from high to low according to their similarity to the first cumulative intent feature. For example, as shown in FIG4 , after calculating the similarity between the first cumulative intent feature and each second-level object, the first-level objects that meet the similarity criteria are sorted from high to low according to their similarity as transmission products, transmission installation tutorials, and transmission repair tutorials. In other words, the second round of search results will be sorted according to their similarity to the first cumulative search intent, and the search results will be displayed to the user. The higher the similarity, the closer it is to the user's first cumulative search intent, and the more likely it is that it is the content the user wants to search for. The higher the ranking, the more convenient it is for the user to find the content they want to search for faster.
[0111] If the second round of search results contains the content the user wants to search for, the user can double-click to open the content and successfully obtain the content the user wants to search for, and the search ends. If the user is still not satisfied with the second round of search results (there is no content that meets the user's search intention), the user continues to interact and conduct the next round of search, continuing to improve the user's search intention until content that meets the user's search intention is found.
[0112] That is, the Mth cumulative intent feature is late interactively fused with the feature of the Nth target object to obtain the final search intent, wherein the Nth target object is the object selected by the user from multiple qualified Nth-level objects, M is a positive integer greater than or equal to 1, and N is a positive integer greater than M; based on the final search intent, the final search result is obtained, and the final search result includes the qualified N+1th-level object corresponding to the Nth target object.
[0113] Optionally, backoff can be supported in each search round. For example, after receiving a backoff command from the user, the search page of the previous round is returned to allow the user to reselect the target object and re-express his or her search intention.
[0114] Optionally, the final search results include card search results (as shown in FIG. 5 a ) and / or expanded search results (as shown in FIG. 5 b ).
[0115] The visual search method provided in this embodiment uses a continuous interaction of visual information in a "click-like" manner to replace the original user text description. This not only reduces the complexity of user operations, but also uses visual information containing more "information" to guide users to refine their search intent. Furthermore, during the process of multimodal information interaction and accumulation, the search recommendation results are continuously adjusted and optimized to improve the effectiveness of the search.
[0116] In another implementation, the final search intent is also related to the first text feature, which is a feature of the query text entered by the user. In other words, in each search round, users can further express their search intent by entering query text, thereby shortening search rounds and finding the search content faster or more accurately.
[0117] For example, in the search results of the Nth search round, the user enters a query text, and the features of the query text are extracted. The features of the cumulative search intent of the search round, the features of the target object, and the features of the query text are late interactively fused. Then, based on the fused features, a search is performed in the content base library or the query recommendation library (the similarity with each object is calculated), and finally the final search result is obtained. The final search result recommends the object with the highest similarity (top 1).
[0118] The visual search method provided in the embodiment of the present application guides user interaction based on the query image and the multi-level query recommendation library, and continuously improves the cumulative intent features until the search is completed (supporting rollback during the process). First, based on the similarity between the query image features and the query recommendation features, the first-level recommendations are selected and sorted from all nodes; then, in the subsequent layers, the child nodes of the user-selected node are returned as candidates for the next layer. Late interaction fusion modeling is performed based on the query image features and the query recommendation text features selected by the user to improve the cumulative intent features. Based on the similarity between the current cumulative intent features and the child node query recommendation text features, the next layer of nodes are pruned and re-sorted online. Multivariate fusion is performed based on the cumulative intent features, the next layer query recommendation features, and the user's additional input text features to retrieve various modal information in the content base library. Late interaction fusion modeling is performed based on the cumulative intent features and the next layer node query recommendation text features to retrieve various modal information in the content base library, and the Top-1 is returned as the query recommendation details of the node. Extended search is performed based on the cumulative intent features (and can be further integrated with the user's additional input text features) to return more content information.
[0119] The visual search method provided in the embodiment of the present application can be applied not only to cloud applications, but also to terminal services, such as search and recommendation of mobile phone albums and galleries. It can further connect terminal-side multimodal information such as videos, pictures, and text messages to realize joint interactive retrieval and recommendation of various modal information on the terminal side.
[0120] Based on the same concept as the embodiment of the aforementioned visual search method, an embodiment of the present application further provides a visual search device 700, which includes units or modules for implementing each step in the visual search method shown in Figures 1-6.
[0121] FIG7 is a schematic diagram of the structure of a visual search device provided in an embodiment of the present application. The device is applied to a computing device. As shown in FIG7 , the visual search device 700 includes at least:
[0122] An acquisition module 701 is used to acquire an image to be searched;
[0123] A cumulative search intent determination module 702 is configured to obtain a first round of search results based on the features of the image to be searched and the features of the first-level objects in the query recommendation library, wherein the first round of search results includes a plurality of first-level objects that meet the criteria;
[0124] The query recommendation library includes N levels of objects, each of the N-1th level objects corresponds to multiple Nth level objects, N is an integer greater than 1, and the objects include text content and / or image content and / or video content and / or audio content;
[0125] Performing late interactive fusion on the features of the image to be searched and the features of the first target object to obtain a first cumulative search intent feature, where the first target object is an object selected by the user from the plurality of first-level objects that meet the criteria;
[0126] The search result determination module 703 is configured to obtain a second round of search results based on the first accumulated search intent feature, where the second round of search results includes a plurality of qualified second-level objects corresponding to the first target object.
[0127] In a possible implementation, a first-level object whose similarity with the image to be searched is greater than a preset threshold is determined as a first-level object that meets the criteria.
[0128] In another possible implementation, the plurality of first-level objects that meet the criteria in the first round of search results are sorted from high to low according to their similarity to the image to be searched.
[0129] In another possible implementation, a second-level object whose similarity with the first cumulative search intent feature is greater than a preset threshold is determined as a qualified second-level object.
[0130] In another possible implementation, multiple qualified second-level objects in the second round of search results are sorted from high to low according to their similarity to the first cumulative search intent feature.
[0131] In another possible implementation, the search result determination module 704 is further configured to perform late interactive fusion on the Mth accumulated intent feature and the feature of the Lth target object to obtain a final search intent, wherein the Lth target object is an object selected by the user from the plurality of Lth-level objects that meet the criteria, M is a positive integer greater than or equal to 1, and L is a positive integer greater than M;
[0132] Based on the final search intent, a final search result is obtained, where the final search result includes an L+1th level object that meets the criteria and corresponds to the Lth target object.
[0133] In another possible implementation, the final search intent is also related to a first text feature, where the first text feature is a feature of the query text input by the user.
[0134] In another possible implementation, the final search results include card search results and / or expanded search results.
[0135] In another possible implementation, the query recommendation library includes information in multiple modalities, and the information in multiple modalities is a tree structure. The nodes of the tree structure represent the objects, and the nodes at different levels of the tree structure represent objects at different levels.
[0136] According to the embodiment of the present application, the visual search device 700 can correspond to executing the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of each module in the visual search device 700 are respectively for implementing the corresponding processes of each method in Figures 1-6. For the sake of brevity, they will not be repeated here.
[0137] An embodiment of the present application also provides a computing device, including at least one processor, a memory, and a communication interface, wherein the processor is configured to execute the methods described in Figures 1-6.
[0138] FIG8 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application.
[0139] As shown in Figure 8, the computing device 800 includes at least one processor 801, a memory 802, and a communication interface 803. The processor 801, memory 802, and communication interface 803 are communicatively connected, either via a wired connection (e.g., a bus) or a wireless connection. The communication interface 803 is used to receive data sent by other devices; the memory 802 stores computer instructions, which the processor 801 executes to perform the visual search method described in the aforementioned method embodiment.
[0140] It should be understood that in the embodiment of the present application, the processor 801 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0141] The memory 802 may include a read-only memory and a random access memory, and provides instructions and data to the processor 801. The memory 802 may also include a nonvolatile random access memory.
[0142] The memory 802 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0143] It should be understood that the computing device 800 according to the embodiment of the present application can execute the method shown in Figures 1-6 in the embodiment of the present application. The detailed description of the implementation of the method is given above and will not be repeated here for the sake of brevity.
[0144] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned visual search method is implemented.
[0145] An embodiment of the present application provides a chip comprising at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is configured to execute the program instructions to implement the visual search method mentioned above.
[0146] An embodiment of the present application provides a computer program or a computer program product, which includes instructions. When the instructions are executed, the computer is caused to execute the visual search method mentioned above.
[0147] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0148] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0149] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.
Claims
1. A visual search method, characterized in that: Get the image to be searched; Obtaining a first round of search results based on the features of the image to be searched and the features of the first-level objects in the query recommendation library, wherein the first round of search results includes a plurality of first-level objects that meet the criteria; The query recommendation library includes N levels of objects, each of the N-1th level objects corresponds to multiple Nth level objects, N is an integer greater than 1, and the objects include text content and / or image content and / or video content and / or audio content; Performing late interactive fusion on the features of the image to be searched and the features of the first target object to obtain a first cumulative search intent feature, where the first target object is an object selected by the user from the plurality of first-level objects that meet the criteria; Based on the first accumulated search intent feature, a second round of search results is obtained, where the second round of search results includes multiple qualified second-level objects corresponding to the first target object.
2. The method according to claim 1, characterized in that A first-level object whose similarity to the image to be searched is greater than a preset threshold is determined as a first-level object that meets the criteria.
3. The method according to claim 1 or 2, characterized in that The plurality of first-level objects that meet the criteria in the first round of search results are sorted from high to low according to their similarity to the image to be searched.
4. The method according to any one of claims 1 to 3, characterized in that The second-level objects whose similarity with the first cumulative search intention feature is greater than a preset threshold are determined as qualified second-level objects.
5. The method according to any one of claims 1 to 4, characterized in that The plurality of qualified second-level objects in the second round of search results are sorted from high to low according to their similarity to the first cumulative search intent feature.
6. The method according to any one of claims 1 to 5, characterized in that Also includes, Performing late interactive fusion on the Mth accumulated intent feature and the feature of the Lth target object to obtain a final search intent, wherein the Lth target object is an object selected by the user from the plurality of Lth-level objects that meet the criteria, M is a positive integer greater than or equal to 1, and L is a positive integer greater than M; Based on the final search intent, a final search result is obtained, where the final search result includes an L+1th level object that meets the criteria and corresponds to the Lth target object.
7. The method according to claim 6, characterized in that The final search intent is also related to a first text feature, where the first text feature is a feature of the query text input by the user.
8. The method according to claim 6 or 7, characterized in that The final search results include card search results and / or expanded search results.
9. The method according to any one of claims 1 to 7, characterized in that The query recommendation library includes information in multiple modalities, and the information in multiple modalities is in a tree structure. The nodes of the tree structure represent the objects, and the nodes at different levels of the tree structure represent objects at different levels.
10. A visual search device, characterized in that: include: An acquisition module, used to acquire the image to be searched; a cumulative search intent determination module, configured to obtain a first round of search results based on features of the image to be searched and features of first-level objects in the query recommendation library, wherein the first round of search results includes a plurality of first-level objects that meet the criteria; The query recommendation library includes N levels of objects, each of the N-1th level objects corresponds to multiple Nth level objects, N is an integer greater than 1, and the objects include text content and / or image content and / or video content and / or audio content; Performing late interactive fusion on the features of the image to be searched and the features of the first target object to obtain a first cumulative search intent feature, where the first target object is an object selected by the user from the plurality of first-level objects that meet the criteria; The search result determination module is used to obtain a second round of search results based on the first cumulative search intention feature, where the second round of search results includes multiple qualified second-level objects corresponding to the first target object.
11. The device according to claim 10, characterized in that A first-level object whose similarity to the image to be searched is greater than a preset threshold is determined as a first-level object that meets the criteria.
12. The device according to claim 10 or 11, characterized in that The plurality of first-level objects that meet the criteria in the first round of search results are sorted from high to low according to their similarity to the image to be searched.
13. The device according to any one of claims 10 to 12, characterized in that: The second-level objects whose similarity with the first cumulative search intention feature is greater than a preset threshold are determined as qualified second-level objects.
14. The device according to any one of claims 10 to 13, characterized in that The plurality of qualified second-level objects in the second round of search results are sorted from high to low according to their similarity to the first cumulative search intent feature.
15. The device according to any one of claims 10 to 14, characterized in that The search result determination module is further configured to perform late interactive fusion of the Mth accumulated intent feature with the feature of the Lth target object to obtain a final search intent, wherein the Lth target object is an object selected by the user from the plurality of Lth-level objects that meet the criteria, M is a positive integer greater than or equal to 1, and L is a positive integer greater than M; Based on the final search intent, a final search result is obtained, where the final search result includes an L+1th level object that meets the criteria and corresponds to the Lth target object.
16. The device according to claim 15, characterized in that The final search intent is also related to a first text feature, where the first text feature is a feature of the query text input by the user.
17. The device according to claim 15 or 16, characterized in that The final search results include card search results and / or expanded search results.
18. The device according to any one of claims 10 to 17, characterized in that The query recommendation library includes information in multiple modalities, and the information in multiple modalities is in a tree structure. The nodes of the tree structure represent the objects, and the nodes at different levels of the tree structure represent objects at different levels.
19. A computing device comprising a memory and a processor, characterized in that: The memory stores executable code, and the processor executes the executable code to implement the method according to any one of claims 1 to 9.
20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Image retrieval processing method, server, client side and storage medium
CN108563792A
Information search method and device, electronic equipment and storage medium
CN113868519A
Data recall method and device
CN114329174A
Image retrieval using interactive natural language dialog
US20190294702A1
