Multi-modal search method for point shopping mall based on vectorization search architecture
By employing a vectorized search architecture and multimodal fusion technology, the problems of search accuracy and personalized recommendations for heterogeneous data in the points mall were solved, enabling efficient and personalized points mall search and recommendations, thereby improving user experience and operational efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BESTTONE HOLDING
- Filing Date
- 2025-12-08
- Publication Date
- 2026-04-17
AI Technical Summary
Traditional points mall search methods struggle to effectively integrate heterogeneous data such as text, images, and audio, resulting in low cross-modal search accuracy and a failure to dynamically optimize sorting based on user profiles, leading to a high mismatch between product recommendations and user needs.
Employing a vectorized search architecture and multimodal fusion technology, this system uses PQ vector compression and hierarchical HNSW indexing, combined with user history and product attributes, to construct user profiles and product feature matrices. Leveraging ELECTRA and LLM deep semantic understanding, it achieves intelligent search and personalized recommendations.
It improves search accuracy and recall, optimizes search response efficiency, provides a personalized experience, enhances user satisfaction and loyalty, and supports fast retrieval and personalized product recommendations in high-concurrency scenarios.
Smart Images
Figure FT_1 
Figure SMS_1
Abstract
Description
Technical Field
[0001] This invention relates to the fields of e-commerce, artificial intelligence, and information retrieval technology, and specifically to a multimodal intelligent search method for an points mall based on a vectorized search architecture. Background Technology
[0002] With the rapid development of points mall operations, users' search needs for points-redeemed goods are becoming increasingly diverse and personalized. Traditional keyword-matching search methods have significant limitations: they struggle to effectively integrate heterogeneous data such as text, images, and audio, resulting in low cross-modal search accuracy; and they fail to dynamically optimize ranking based on user profiles, leading to a high mismatch between product recommendations and user needs. Therefore, there is an urgent need to find smarter and more efficient search methods to improve user experience and shopping efficiency.
[0003] In recent years, artificial intelligence has made significant progress in the vertical application of search. The development of technologies such as deep learning and natural language processing, especially the rise of large language models (LLMs), has demonstrated powerful potential in deeply understanding user intent, processing complex information, and achieving accurate matching. Meanwhile, vectorized search technology, by uniformly mapping heterogeneous data to a high-dimensional vector space and utilizing near nearest neighbor (ANN) indexes (such as HNSW) to achieve cross-modal semantic alignment and efficient retrieval, provides a new technical path for solving the aforementioned problems. Summary of the Invention
[0004] In response to the above situation and to overcome the shortcomings of existing technologies, this invention is based on a vectorized search architecture and multimodal fusion to achieve intelligent search functionality. By combining user history and product attributes, it provides personalized search results, simplifies the search process, reduces filtering costs, and enhances the convenience of points-based consumption.
[0005] Breaking through the bottleneck of multimodal search: By using PQ vector compression technology and hierarchical HNSW indexing, storage and computing overhead are optimized to support high-concurrency scenarios with an average daily QPS of 20 million, while reducing server resource consumption.
[0006] Achieve dynamic optimization of e-commerce operations: Build a mechanism for continuous feedback of user behavior data, update model parameters through incremental learning and iteration, adapt to changes in user preferences, eliminate feature decay caused by data drift, and enhance the long-term competitiveness of the system; The technical solution of this invention is achieved through the collaborative efforts of six core modules, the functions and processes of which are as follows: Multimodal Information Fusion Module (S01): This module focuses on fusing various modal information such as text, images, audio, and video uploaded by users during searches. It integrates RTC (Real-Time Communication) and ITC (Information Transmission Control) technologies to efficiently fuse and transmit multimodal information through streaming, shortening transmission time. It cleans, labels, and preprocesses the multimodal data to ensure comparability and consistency; extracts and integrates features from each modality, performs time alignment and spatial synchronization, and generates a unified semantic representation (such as descriptive text or fusion vectors) through fusion strategies to comprehensively understand product and user needs. Model Factors Factor Explanation Text segmentation Break down the user-input sentence into meaningful lexical units. Part-of-speech tagging Part-of-speech tagging involves classifying these lexical units, such as nouns, verbs, and adjectives. Syntactic analysis Syntactic analysis reveals the structural relationships between words. Through these processes, the module can extract key information, such as the type of product the user wants to find, specific brands, price ranges, or other shopping preferences. Combined with contextual search information within the user's current search action, the module can more accurately infer the user's search intent. For example, if a user frequently searches for a certain type of product and uses related keywords in the current search, the module can infer that the user may be looking for new options or related products within the same category.
[0007] Data information integration module (S03): As the core data processing hub, it integrates product data and user profiles in a structured manner and embeds a vectorized search architecture to achieve efficient semantic retrieval.
[0008] Multimodal data vectorization encoding for products: Multi-dimensional feature extraction is performed on the structured data (name, description, attributes, redemption points, etc.) and unstructured data (images, videos, audio, etc.) of all products in the points mall; heterogeneous data are uniformly mapped to a 1280-dimensional vector space through an embedded model group (ELECTRA for text processing / ResNet-50 for image processing / WaveNet for audio processing) to construct a semantic feature matrix for products.
[0009] User profile construction and vectorization: User information, historical records, and purchase behavior data are linked by user ID. The resulting data is standardized to ensure that data from different sources have a unified format and measurement standards. Data deduplication, formatting correction, and logical error correction are performed. Mean averaging is used for data completion. The above data information is cleaned, integrated, and aggregated for analysis to ensure accuracy, uniqueness, and reliability. Finally, a pre-trained model (such as BERT) is used to encode the textual descriptions of user profiles (summarizing key features) into vectors (such as a 1280-dimensional user preference vector).
[0010] Vector Index Construction and Maintenance: A multi-layer graph index is constructed based on the Hierarchical Navigable Small World (HNSW) algorithm, enabling millisecond-level retrieval of a library of hundreds of millions of vectors. Newly added products are periodically vectorized, and their vectors are injected into the HNSW index in real time to ensure rapid retrieval. Simultaneously, based on user behavior feedback captured by the SO5 module (such as negative samples of clicks on "not purchased"), the embedding model parameters are fine-tuned weekly to maintain consistency between the feature space and user preferences, mitigating data drift.
[0011] This module requires regular optimization and improvement, including periodic data quality assessments, continuous algorithm optimization, and expansion of user-related data sources to ensure the accuracy and completeness of all data. During data integration and user profile building, strict adherence to privacy regulations is maintained to ensure the legality and security of user data.
[0012] Intelligent Search Module (S04): The main function of this module is to enable efficient and accurate searching of multi-module information input by the user. First, after capturing the user's query intent (Module 2), this module quickly filters out products related to the user's query from the multi-module product information database of the points mall (Module 3). Second, it combines these products with a search ranking model to ensure that products that better match the user's preferences and needs are displayed first. The search ranking model is as follows:
[0013] The search ranking model greatly improves the targeting and user experience of the search. The intelligent search module adopts a product attribute matching algorithm and uses a multi-level search strategy to help users quickly find their favorite products in milliseconds. It can also provide intelligent recommendations when users do not have a specific goal, thereby stimulating users' shopping interest and improving the overall conversion rate and user satisfaction of the points mall.
[0014] Finally, the intelligent search module of the points mall is continuously optimized. Based on user click-through rate and order conversion rate as evaluation criteria, and combined with the L2R ranking learning algorithm, the weights of each factor in the search ranking model are continuously optimized and adjusted to ensure that the products that best meet user expectations can always be ranked at the top of the search results, bringing users a more convenient and efficient shopping experience.
[0015] User Search Data Maintenance Module (S05): This module's main function is to collect and record user behavior data during the search process, providing information for the updates and optimizations of other modules. First, it records the keywords used in each user search in detail. These keywords reflect not only the user's immediate needs but also their interest trends. Through the accumulation and analysis of keywords, the module can construct a semantic network of user behavior, laying the foundation for understanding users' deeper needs. Second, it tracks user click behavior, including which products the user clicked, the order of clicks, and the frequency of clicks. This data provides a basis for adjusting the weights of various factors in the search ranking model. Furthermore, this module also needs to collect data such as the time users spend on product pages and their post-search purchasing behavior, mining multi-dimensional data to provide rich material for building user profiles.
[0016] Purchase Prediction Module (S06): This module's main function is to provide an analysis tool for the points mall. It performs data mining processing on diverse data such as user profiles, search behaviors, and purchase records generated throughout the intelligent search process. Combining algorithms such as regression analysis and time series analysis, it accurately predicts the purchase of products in the multi-modal product information database. The output is each user's probability score for purchasing products in the mall, which helps the points mall to carry out personalized marketing, product recommendations, inventory management, and user segmentation.
[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. Significantly improved search accuracy and recall: Through multimodal fusion and ELECTRA+LLM deep semantic understanding, it comprehensively captures complex user intent; the vectorized search architecture achieves cross-modal semantic alignment, solves the bottleneck of traditional keyword matching, and improves the accuracy of heterogeneous data retrieval.
[0018] 2. Optimized search response efficiency: HNSW hierarchical indexing combined with PQ vector compression technology enables millisecond-level retrieval of hundreds of millions of products, supporting high-concurrency scenarios with an average daily QPS of 20 million, and reducing the consumption of computing resources such as GPUs.
[0019] 3. Enhanced Personalized Experience: The L2R ranking model, which integrates user profiles, provides customized search results and recommendations, improving user satisfaction, dwell time, and purchase conversion rates.
[0020] 4. System Adaptation and Continuous Evolution: The incremental learning mechanism driven by user behavior data enables the model to quickly respond to changes in user preferences and product updates, alleviates data drift problems, and ensures the system's optimal performance in the long term.
[0021] 5. Intelligent operational decision-making: Abundant user behavior data supports the mall in optimizing product layout, improving inventory management efficiency, and formulating precise marketing strategies, thereby improving overall operational efficiency and revenue.
[0022] 6. Enhance user satisfaction and loyalty: Accurate search results and personalized recommendations enhance the user shopping experience, increase user loyalty to the points mall, and promote repeat purchases. Attached Figure Description
[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.
[0024] In the attached diagram: Figure 1 This is a block diagram of the digital employee collaborative execution system based on the points mall of the present invention. Detailed Implementation
[0025] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0026] In order to solve the problems in the background art, please refer to the following: Figure 1 ; Users can submit search requests containing one or more modalities such as text, images, audio, and video through the points mall; The multimodal information fusion module (S01) receives multimodal input, uses RTC / ITC technology for efficient transmission, performs cleaning, preprocessing, feature extraction and fusion, and generates a unified semantic representation; The semantic understanding module (S02) performs word segmentation, part-of-speech tagging, syntactic analysis, and intent recognition on the unified semantic representation (or directly processes text input) to generate a user query intent vector; The data information integration module (S03) provides a completed product multimodal feature vector library, user profile vector library, and HNSW vector index to support efficient retrieval; The intelligent search module (S04) takes the user's query intent vector as input vector index, obtains a set of candidate products through ANN retrieval, and after fine ranking by the L2R ranking model, outputs the sorted product list to the user; The user search data maintenance module (S05) tracks and records user behavior data such as keywords, clicks, dwell time, and purchases throughout the process, stores it, and feeds it back to the data information integration module (for model training) and the intelligent search module (for ranking optimization). The purchase prediction module (S06) predicts the probability of user purchases based on accumulated user and product data, providing data support for mall operation decisions.
Claims
1. A method for multimodal search of an points mall based on a vectorized search architecture, characterized in that, Includes the following steps: The system receives multimodal search input from users; it fuses the multimodal search input to generate a unified semantic representation; it performs semantic understanding on the unified semantic representation or text input based on a language model and the ELECTRA algorithm to generate a user query intent vector; it constructs and maintains a multimodal feature vector library for products and a user profile vector library for the points mall, where product feature vectors are generated through an embedding model group (ELECTRA for text processing / ResNet-50 for image processing / WaveNet for audio processing); it constructs and dynamically updates a vector index based on the hierarchical navigable small world (HNSW) algorithm; it uses the user query intent vector to perform an approximate nearest neighbor (ANN) search in the vector index to obtain a candidate product set; it applies a learning to rank (L2R) model to perform personalized ranking on the candidate product set, whereby the L2R model is calculated based on user profile matching degree, product-query relevance, product title-search term matching degree, product sales rating, product freshness, and product-specific weight factors (including points redemption rate and inventory); and it outputs a ranked product list. Collect user search behavior data (including keywords, click behavior, dwell time, and purchase history); based on the behavior data, periodically fine-tune the parameters of the embedded model group and update the vector index through incremental learning.
2. The method according to claim 1, characterized in that, The fusion processing of the multimodal search input is achieved through a multimodal information fusion module. This module integrates RTC (Real-Time Communication) technology and ITC (Information Transmission Control) technology to clean, label, preprocess, extract features, align time, and synchronize space for multimodal data. Through fusion strategies, it generates a unified semantic representation, thereby improving the accuracy and personalization of search results.
3. The method according to claim 1, characterized in that, The semantic understanding is achieved through a semantic understanding module, which extracts key information through text segmentation, part-of-speech tagging, and syntactic analysis. Combined with the user's search context and historical behavior, it accurately infers the user's search intent. The text processing relies on the ELECTRA algorithm and the natural language processing capabilities of LLM.
4. The method according to claim 1, characterized in that, The construction of the product multimodal feature vector library and user profile vector library is achieved through a data information integration module: multi-dimensional feature extraction and vectorization encoding are performed on the structured data (name, description, attributes, redemption points, etc.) and unstructured data (images, videos, audio, etc.) of the products to generate a 1280-dimensional product semantic feature vector; user basic information, historical records, and purchase behavior data are standardized, cleaned and integrated, and encoded into a 1280-dimensional user preference vector through a pre-trained BERT model.
5. The method according to claim 1, characterized in that, The collection of user behavior data is achieved through the user search data maintenance module. This module tracks, records, and mines the user search behavior data throughout the entire process, providing data support for the optimization of the product multimodal feature vector library and the user profile vector library, as well as the weight adjustment of the L2R ranking model.
6. The method according to claim 1, characterized in that, It also includes a purchase prediction module, which uses data mining based on user profiles, search behavior, purchase records and product feature data, combined with algorithms such as regression analysis and time series analysis, to output a user's probability score of purchasing products in the mall, supporting personalized marketing, product recommendations, inventory management and user segmentation operations.
7. The method according to claim 1, characterized in that, The original 2048-dimensional vector is compressed to 64 dimensions using product quantization (PQ) technology, reducing storage and computational overhead. The embedded model group adopts a weekly incremental training mechanism, using newly added product data and user behavior feedback to fine-tune parameters and maintain consistency between the feature space and user preferences.