Construction and retrieval method of dynamic vector knowledge base based on multimodal large model

Abstract

The present invention belongs to the field of knowledge retrieval technology. The present invention discloses a dynamic vector knowledge base construction and retrieval method based on a multimodal large model, comprising: obtaining a multi-source heterogeneous modal data set, and performing preprocessing and modality normalization processing on the data set to obtain a standardized multimodal data set; using a pre-trained multimodal large model to perform feature extraction and generate semantic vector representations to construct a multimodal knowledge vector set; performing semantic association analysis and hierarchical clustering on the multimodal knowledge vector set to construct a structured vector knowledge base; performing semantic similarity calculation and relationship modeling on the vector knowledge base to form a vector relationship network; performing intent analysis and vector representation based on mixed modal query information input by a user, and combining the vector relationship network to achieve efficient similarity retrieval; performing dynamic optimization through user feedback to achieve personalized retrieval result adjustment, effectively solving the limitations of traditional retrieval systems when processing multimodal data.

Description

Technical Field

[0001] The present invention relates to the field of knowledge retrieval technology, and more specifically, to a method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model. Background Art

[0002] In the field of knowledge retrieval and data management, multimodal information processing, as a core technical capability, is of great significance for achieving comprehensive and accurate knowledge acquisition. With the rapid development of information technology, users' requirements for the intelligence, personalization and efficiency of knowledge retrieval systems are increasing, and traditional knowledge retrieval and management models are gradually exposing many problems.

[0003] Compared with existing technologies, traditional knowledge retrieval systems mainly rely on single-modal data processing, which is difficult to handle multi-source data forms such as text, images, audio and video, resulting in one-sided and incomplete retrieval results. Moreover, different modal data are stored in a scattered manner and the representation forms are not unified, making it difficult to conduct effective integration and correlation analysis. Moreover, due to the lack of a unified semantic representation framework, cross-modal information retrieval faces the semantic gap problem, resulting in low retrieval accuracy and inability to meet users' complex and diverse query needs.

[0004] Traditional methods have a relatively simple modeling of knowledge semantic relationships, lack deep semantic understanding and vectorized representation, and have difficulty capturing implicit associations between knowledge. As a result, the retrieval system can only perform simple keyword matching and cannot understand the true intention of the user's query, which also affects the relevance and comprehensiveness of the retrieval results. In terms of knowledge organization structure, traditional methods have difficulty in effectively processing large-scale heterogeneous data and lack efficient indexing mechanisms and optimized storage strategies, resulting in low retrieval efficiency and increased system burden. They cannot achieve a personalized retrieval experience and it is difficult to dynamically adjust the retrieval strategy based on user feedback, resulting in low user satisfaction.

[0005] In view of this, the present invention proposes a dynamic vector knowledge base construction and retrieval method based on a multimodal large model to solve the above problems. Summary of the Invention

[0006] In order to overcome the above-mentioned defects of the prior art and achieve the above-mentioned objectives, the present invention provides the following technical solutions:

[0007] The dynamic vector knowledge base construction and retrieval method based on the multimodal large model includes:

[0008] Step 1: Obtain a multi-source heterogeneous modal dataset, and perform preprocessing and modal normalization on the obtained multi-source heterogeneous modal dataset to obtain the corresponding standardized multi-modal dataset;

[0009] Step 2: Based on the pre-built multimodal large model, feature extraction and semantic vector representation generation are performed on the standardized multimodal dataset to obtain the corresponding multimodal knowledge vector set;

[0010] Step 3: Perform semantic association analysis and hierarchical clustering on the obtained multimodal knowledge vector set to construct a vector knowledge graph structure, and then build a structured vector knowledge base based on the vector knowledge graph structure;

[0011] Step 4: Calculate semantic similarity and model the relationship between vectors on the obtained vector knowledge base to obtain the corresponding vector relationship network;

[0012] Step 5: Perform multimodal query understanding and intent analysis on the mixed-modal query information input by the user to obtain the corresponding query vector representation; and combine it with the vector relationship network to perform efficient similarity search to obtain the corresponding initial search result set;

[0013] Step 6: Perform dynamic feedback optimization on the initial search result set to obtain an optimized search result set.

[0014] Furthermore, a data acquisition unit is provided, which is used to collect and integrate different modal data sets from multiple data sources to obtain corresponding multi-source heterogeneous modal data sets; the multi-source heterogeneous modal data sets include text data, image data, audio data and video data.

[0015] Furthermore, the process of obtaining a standardized multimodal dataset includes:

[0016] Perform modality identification and format conversion on the obtained multi-source heterogeneous modal dataset to obtain the corresponding initial modal dataset;

[0017] Perform data cleaning and outlier detection on the obtained initial modal data set to obtain the corresponding purified modal data set;

[0018] Performing data enhancement and normalization processing on the obtained purified modal dataset to obtain the corresponding enhanced modal dataset;

[0019] Perform cross-modal alignment and time synchronization on the obtained enhanced modality dataset to obtain the corresponding multimodal aligned dataset;

[0020] The obtained multimodal aligned dataset is subjected to feature normalization and dimension consistency processing to obtain the corresponding standardized multimodal dataset.

[0021] Furthermore, the process of acquiring the multimodal knowledge vector set includes:

[0022] Based on known modality types, pre-trained large model components suitable for different modal data are selected; the selected pre-trained large model components are integrated and a corresponding multimodal large model is constructed based on them;

[0023] Based on the constructed multimodal large model, feature extraction is performed on the corresponding standardized multimodal dataset to obtain the corresponding modal feature matrix;

[0024] Performing cross-modal attention fusion and semantic alignment on the modal feature matrix to obtain a fused feature representation in a unified semantic space;

[0025] Perform knowledge distillation and feature compression on the obtained fusion feature representation to obtain the corresponding efficient dense vector;

[0026] The obtained efficient dense vectors are subjected to semantic similarity verification and quality assessment to obtain the corresponding multimodal knowledge vector set.

[0027] Furthermore, the construction process of the vector knowledge graph structure includes:

[0028] Perform similarity calculation and nearest neighbor analysis on the obtained multimodal knowledge vector set, and build the corresponding initial semantic association network based on it;

[0029] Based on the community detection algorithm, the constructed initial semantic association network is subjected to semantic clustering identification to obtain the corresponding semantic community structure;

[0030] Based on the obtained semantic community structure, a multi-level semantic classification system is constructed by applying a hierarchical clustering algorithm to obtain a corresponding hierarchical semantic tree; the node relationships of the obtained hierarchical semantic tree are enhanced, and cross-level and cross-branch semantic associations are added to obtain a corresponding vector knowledge graph structure.

[0031] Furthermore, the process of constructing the vector knowledge base includes:

[0032] Use a multi-level indexing mechanism to organize efficient dense vectors and design a corresponding hierarchical index structure based on it; the multi-level indexing mechanism can include inverted index and tree index;

[0033] The obtained efficient dense vector is compressed and encoded to obtain a quantized vector with low storage cost; and a corresponding quantization-restoration conversion model is constructed based on the vector;

[0034] Based on the vector knowledge graph structure, the hierarchical index structure and quantization-reduction conversion model between vectors are integrated, and a multi-level vector knowledge base is constructed based on it.

[0035] Furthermore, the construction process of the vector relationship network includes:

[0036] Perform semantic similarity calculation on the obtained vector knowledge base to obtain the semantic similarity between different vectors in the vector knowledge base; and construct a corresponding similarity matrix based on the semantic similarity;

[0037] Based on the obtained similarity matrix, high-density semantic association regions are identified by applying threshold filtering and density clustering to obtain corresponding semantic association clusters; relationship types are identified and attributes are labeled on the obtained semantic association clusters to obtain corresponding semantic relationship triples;

[0038] Constructing a relational graph structure between vectors based on the semantic relation triples, and performing graph embedding optimization to obtain a corresponding initial vector relational network; and performing relational reasoning and link prediction on the initial vector relational network to obtain a corresponding enhanced vector relational network;

[0039] The obtained enhanced vector relationship network is subjected to weight optimization and path analysis to obtain a corresponding vector relationship network.

[0040] Furthermore, the process of obtaining the query vector representation includes:

[0041] Setting a user interaction unit and obtaining mixed-modal query information input by a target user based on the user interaction unit;

[0042] Perform modal separation and identification on the input mixed-modal query information; obtain corresponding modal-specific query components; normalize the obtained modal-specific query components to obtain corresponding standardized query representations;

[0043] The obtained standardized query representation is vectorized and mapped using a multimodal large model, and converted into a vector representation in a unified semantic space to obtain the corresponding query initial vector;

[0044] Based on the constructed vector relationship network, the input mixed-modal query information is understood and expanded to obtain the corresponding enhanced intent description; based on this, the obtained initial query vector is reweighted and adjusted to obtain the enhanced intent query vector;

[0045] Perform contextual analysis on the query vector with enhanced intent and combine it with the user's historical query behavior to obtain the corresponding query vector representation.

[0046] Furthermore, the process of obtaining the initial search result set includes:

[0047] Design a multi-level search strategy based on the constructed hierarchical index structure, and perform a coarse-grained search based on the obtained query vector representation to obtain a corresponding candidate result set; and filter the candidate result set based on pre-set metadata filtering conditions to obtain a streamlined candidate set; the metadata filtering conditions include time range, source restriction, and modality type;

[0048] Performing fine-grained similarity calculation on the reduced candidate set to obtain an accurate similarity score; and sorting the reduced candidate set based on the similarity score to obtain a corresponding ordered result list;

[0049] Based on the ordered result list, the distribution and correlation of data corresponding to different modal types in the retrieval results are obtained, and multimodal fusion and complementary enhancement are performed based on them to obtain an initial retrieval result set with modal balance.

[0050] Furthermore, the process of obtaining the optimized search result set includes:

[0051] Feedback the obtained initial search results to the target user, and collect the user's feedback information and user interaction behavior on the initial search results based on the user interaction unit;

[0052] Based on the feedback information corresponding to the target user, the corresponding initial search results are marked to obtain the search content selected by the user; and by analyzing the implicit association pattern between it and the input mixed-modal query information, the corresponding query-result association graph is constructed;

[0053] Integrate the historical feedback information, historical user interaction behavior, and historical user query behavior obtained; clean and standardize them to form structured user preference data;

[0054] Based on user preference data and query-result association graph, the designed multi-level retrieval strategy is adjusted using reinforcement learning methods to form a personalized retrieval model;

[0055] Obtain initial search results and user preference data to optimize threshold parameters; obtain corresponding adaptive threshold strategies;

[0056] The personalized retrieval model and the adaptive threshold strategy are combined to re-rank and optimize the initial retrieval result set to obtain the corresponding optimized retrieval result set.

[0057] The technical effects and advantages of the dynamic vector knowledge base construction and retrieval method based on the multimodal large model of the present invention are as follows:

[0058] 1. By pre-training a large multimodal model and using a unified semantic space mapping, the collected multi-source heterogeneous modal datasets are pre-processed, feature extracted, and semantic vector generated. This achieves unified representation and semantic interoperability of data from different modalities, resulting in a high-quality multimodal knowledge vector set. This not only accurately captures cross-modal semantic associations but also effectively eliminates the semantic gap, providing strong support for complex queries. This overcomes the limitations of traditional single-modality retrieval and improves the comprehensiveness and accuracy of retrieval results.

[0059] 2. By employing algorithms such as hierarchical clustering and graph structure modeling, semantic association analysis and structured organization are performed on multimodal knowledge vector sets. A semantically rich vector knowledge graph is constructed, a multi-level indexing mechanism is designed, and a product quantization compression storage strategy is provided to form an efficient structured vector knowledge base. Vector relationship network modeling is used to identify potential semantic associations, providing key support for efficient retrieval. This effectively addresses the storage and retrieval efficiency issues of traditional systems in processing massive heterogeneous data, significantly reducing system resource consumption and improving retrieval response speed.

[0060] 3. By combining user feedback with a reinforcement learning mechanism, dynamic optimization of personalized retrieval is achieved, forming a highly personalized retrieval experience. This not only accurately understands the user's true query intent, but also continuously learns user preferences and dynamically optimizes retrieval results, solving the problem of traditional retrieval systems lacking personalization capabilities and significantly improving user satisfaction and system usage value. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model according to the present invention;

[0062] Figure 2 Schematic diagram of the dynamic vector knowledge base construction and retrieval system based on the multimodal large model of the present invention. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0064] Example 1

[0065] See also Figure 1 As shown, the method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model described in this embodiment includes:

[0066] Step 1: Obtain a multi-source heterogeneous modal dataset, and perform preprocessing and modal normalization on the obtained multi-source heterogeneous modal dataset to obtain the corresponding standardized multi-modal dataset;

[0067] Step 2: Based on the pre-built multimodal large model, feature extraction and semantic vector representation generation are performed on the standardized multimodal dataset to obtain the corresponding multimodal knowledge vector set;

[0068] Step 3: Perform semantic association analysis and hierarchical clustering on the obtained multimodal knowledge vector set to construct a vector knowledge graph structure, and then build a structured vector knowledge base based on the vector knowledge graph structure;

[0069] Step 4: Calculate semantic similarity and model the relationship between vectors on the obtained vector knowledge base to obtain the corresponding vector relationship network;

[0070] Step 5: Perform multimodal query understanding and intent analysis on the mixed-modal query information input by the user to obtain the corresponding query vector representation; and combine it with the vector relationship network to perform efficient similarity search to obtain the corresponding initial search result set;

[0071] Step 6: Perform dynamic feedback optimization on the initial search result set to obtain an optimized search result set.

[0072] It should be further explained that, in the specific implementation process, the acquisition process of the standardized multimodal dataset includes:

[0073] A data acquisition unit is set up, which is used to collect and integrate different modal data sets from multiple data sources to obtain corresponding multi-source heterogeneous modal data sets; the multi-source heterogeneous modal data sets include text data, image data, audio data and video data; the data sources include publicly available databases and web pages, etc.; for example: collecting text corpora from public academic databases, collecting high-quality image data from open dataset platforms, obtaining various sound records from audio libraries, and capturing diverse video samples from video content platforms; at the same time, using web crawler technology to obtain multimodal content from Internet web pages, and integrating with third-party services through API interfaces to realize automated collection of multi-source data;

[0074] Perform modality identification and format conversion on the obtained multi-source heterogeneous modal data set to obtain the corresponding initial modality data set; modality identification refers to extracting content features from the modal data in the corresponding multi-source heterogeneous modal data set through a content feature recognition algorithm, and identifying the modality type in combination with format metadata such as file extension and MIME type; modality types include visual modality, textual modality, auditory modality and other composite modalities; format conversion refers to converting the collected multi-source heterogeneous modal data into preset format requirements based on the identified modality type; for example: for image data in the visual modality, it can be converted into a unified PNG or JPEG format, and the image resolution can be unified; for text data in the text modality, it can be converted into UTF-8 encoded TXT or JSON format;

[0075] The obtained initial modal dataset is cleaned and outlier detected to obtain the corresponding purified modal dataset. Data cleaning includes processes such as removing duplicate data, repairing damaged data, and filling missing values. In other words, data cleaning is used to remove or repair data that contains errors, is incomplete, or is irrelevant. For example, text data is spelled correctly, HTML tags are removed, and meaningless symbols are deleted. Image data is blurred and overly blurred images are removed. Audio data is noise-checked and audio clips with low signal-to-noise ratios are filtered. Outlier detection uses statistical methods to identify and process data samples that deviate from the normal distribution. The statistical methods used include Z-score and IQR methods. Furthermore, through data cleaning and anomaly detection, a purified modal dataset with higher data quality is obtained.

[0076] The obtained purified modal dataset is subjected to data enhancement and normalization processing to obtain the corresponding enhanced modal dataset; data enhancement refers to enhancing the modal data in the purified modal dataset by adopting different enhancement methods based on the modality type; for example, text data is enhanced by using synonym replacement, syntax tree transformation, back translation and other methods; image data is enhanced by using rotation, flipping, scaling, cropping, color conversion and other methods; normalization processing refers to unifying the purified modal dataset after data enhancement to the corresponding numerical range to reduce the deviation of data distribution and the heterogeneity within the modality, and obtain the corresponding enhanced modal dataset; for example, the image pixel value is normalized to the interval [0, 1];

[0077] The obtained enhanced modality dataset is subjected to cross-modal alignment and temporal synchronization to obtain the corresponding multimodal aligned dataset. Cross-modal alignment refers to establishing the correspondence between different modal data within the enhanced modality dataset, for example, the correspondence between video frames and corresponding audio clips, and between images and descriptive text. Temporal synchronization is used to ensure that modal data with time attributes are unified in the time dimension. The temporal synchronization process is achieved by using timestamp matching and interpolation algorithms. Furthermore, through cross-modal alignment and temporal synchronization, the semantic consistency of different modal data is ensured.

[0078] Furthermore, the obtained multimodal aligned dataset is subjected to feature normalization and dimensional consistency processing to obtain the corresponding standardized multimodal dataset; feature normalization refers to scaling different modal data to a uniform magnitude, and commonly used methods include Z-score normalization and Min-Max scaling; dimensional consistency processing is used to ensure that different modal data have the same or compatible dimensional structure during the data processing process, providing a unified interface for subsequent model processing.

[0079] It should be further explained that, in the specific implementation process, the process of obtaining the multimodal knowledge vector set includes:

[0080] Based on the known modality type, select pre-trained large model components suitable for different modal data. Pre-trained large model components include pre-trained language models, audio models, time series vision models, etc. For example, for text data, you can choose pre-trained language models such as BERT or RoBERTa; for image data, you can choose pre-trained vision models such as ResNet, ViT, CLIP; for audio data, you can choose pre-trained audio models such as Wav2Vec and HuBERT; for video data, you can choose time series vision models such as VideoSwin and TimeSformer;

[0081] Then, the selected pre-trained large model components are integrated and a corresponding multimodal large model is constructed based on them. The multimodal large model can be used to extract features from modal data of different modal types. The corresponding multimodal large model training process must ensure the interface compatibility and fusibility of the output representations between the various pre-trained large model components.

[0082] Then, based on the constructed multimodal large model, feature extraction is performed on the corresponding standardized multimodal dataset to obtain the corresponding modal feature matrix; the modal feature matrix refers to the set of high-dimensional feature representations of each modal data after being extracted by the corresponding pre-trained large model component; it should be further explained that in the feature extraction process, corresponding modal preprocessing strategies need to be adopted for different modal types; for example: text data is converted into model input format through steps such as word segmentation and encoding; image data is prepared as visual model input through steps such as scaling and normalization; audio data is converted into a form that can be processed by the audio model through spectral analysis, MFCC feature extraction, etc.

[0083] Cross-modal attention fusion and semantic alignment are performed on the modal feature matrix to obtain a fused feature representation in a unified semantic space. The fused feature representation is a comprehensive feature representation containing multimodal feature information in a unified semantic space. Cross-modal attention fusion refers to the use of a multimodal attention mechanism or a Transformer architecture to make features between different modal data pay attention to each other in order to capture the semantic associations between cross-modal data. Semantic alignment refers to mapping features of different modal data into the same semantic space through algorithms such as contrastive learning or shared projection space to ensure that semantically similar content is close to each other in the corresponding semantic space.

[0084] Then, the obtained fusion feature representation is subjected to knowledge distillation and feature compression to obtain the corresponding efficient dense vector. Knowledge distillation refers to migrating the extracted complex fusion feature representation into a simpler and lighter feature representation. Feature compression uses methods such as principal component analysis (PCA), autoencoders, or low-rank approximation to reduce vector dimensions and computational complexity while retaining key semantic information and integrity. The dimension of the vector after feature compression is usually set within a fixed range to balance expressiveness and computational efficiency.

[0085] Perform semantic similarity verification and quality assessment on the obtained efficient dense vectors to obtain the corresponding multimodal knowledge vector set; the multimodal knowledge vector set includes the identifier, vector value, metadata and other information of each efficient dense vector;

[0086] Semantic similarity verification refers to the discriminative ability of the high-efficiency dense vectors obtained by testing preset semantic similarity tasks. Commonly used semantic similarity tasks include similar document retrieval and image-text matching. Quality assessment uses methods such as the silhouette coefficient and semantic preservation metric to evaluate the distribution quality and effectiveness of the corresponding high-efficiency dense vectors in the semantic space. The silhouette coefficient represents the average Euclidean distance between the high-efficiency dense vector and other vectors in the corresponding semantic space. The semantic preservation metric is used to quantify the degree of semantic information retention during the vector representation of the corresponding modality data in the semantic space.

[0087] Furthermore, high-quality and efficient dense vectors are screened through semantic similarity verification and quality assessment to form a corresponding multimodal knowledge vector set.

[0088] It should be further explained that, in the specific implementation process, the construction process of the structured vector knowledge base includes:

[0089] The obtained multimodal knowledge vector set is subjected to similarity calculation and nearest neighbor analysis, and a corresponding initial semantic association network is constructed based on it. Similarity calculation obtains the semantic similarity between different efficient dense vectors in the multimodal knowledge vector set by using measurement methods such as cosine similarity, Euclidean distance or inner product. Nearest neighbor analysis uses the k-nearest neighbor algorithm to identify other efficient dense vectors that are most similar to the corresponding efficient dense vector, divides them into a vector pair, and simultaneously constructs a corresponding initial semantic association network based on them. The initial semantic association network is a weighted graph structure consisting of nodes and edges. Nodes represent corresponding efficient dense vectors; edges represent semantic relationships, and edge weights are used to reflect the similarity strength of semantic similarity.

[0090] The constructed initial semantic association network is then subjected to semantic cluster identification based on a community detection algorithm to obtain the corresponding semantic community structure. Community detection algorithms, such as the Louvain algorithm, the InfoMap algorithm, or the label propagation algorithm, are used to discover closely connected groups of nodes in a graph structure. A semantic cluster refers to a collection of vectors with high semantic similarity and close connections. A semantic community structure consists of multiple semantic clusters and the relationships between them.

[0091] Based on the obtained semantic community structure, a multi-level semantic classification system is constructed by applying a hierarchical clustering algorithm to obtain a corresponding hierarchical semantic tree. The hierarchical clustering algorithm adopts a bottom-up agglomerative clustering method or a top-down divisive clustering method, and gradually merges or splits clusters according to semantic distance. In this way, the corresponding semantic community structure is organized into a tree structure. The multi-level semantic classification system refers to semantic categories containing multiple levels of abstraction, forming a hierarchical relationship from specific concepts to abstract concepts.

[0092] The obtained hierarchical semantic tree is enhanced with node relationships, and cross-level and cross-branch semantic associations are added to obtain the corresponding vector knowledge graph structure. The vector knowledge graph structure is a complex network containing multiple semantic relationships. It not only retains the classification information of the hierarchical structure but also contains rich horizontal semantic associations. Node relationship enhancement refers to adding non-hierarchical associations on the basis of the hierarchical semantic tree based on information such as semantic similarity, co-occurrence frequency between vector representations, or external knowledge bases.

[0093] A multi-level indexing mechanism is used to organize efficient dense vectors, and a corresponding hierarchical index structure is designed based on this. The multi-level indexing mechanism can include inverted indexes, tree indexes, etc. The design process of the hierarchical index structure can be implemented using algorithms such as HNSW or IVF. The semantic space where the efficient dense vectors are located is divided into multiple subspaces, and a multi-level index hierarchy is constructed from coarse-grained to fine-grained. At the same time, at each index level, an appropriate index structure is designed based on the distribution characteristics of the efficient dense vectors, such as tree indexes, graph indexes, or hybrid indexes, to form a vector index system that can be quickly retrieved.

[0094] At the same time, the product quantization algorithm is used to compress and encode efficient dense vectors to reduce storage space requirements and obtain quantized vectors with low storage costs. The product quantization algorithm significantly reduces storage space requirements by decomposing high-dimensional vectors into multiple low-dimensional subspaces and independently quantizing each subspace. For example, a 256-dimensional vector can be divided into 16 16-dimensional subvectors, each of which is independently quantized to an index in the 256 codebook vectors, compressing the original vector to 1 / 8 of its original size.

[0095] Furthermore, the mapping relationship before and after quantization is preserved, and based on it, a codebook and reconstruction parameters are established to support fast conversion between quantized vectors and original vectors, and obtain the corresponding quantization-restoration conversion model;

[0096] Based on the vector knowledge graph structure, the hierarchical index structure and quantization-reduction conversion model between vectors are integrated, and a multi-level vector knowledge base is constructed based on it; the vector knowledge base contains the semantic relationship structure of efficient dense vectors, efficient retrieval mechanism and storage optimization strategy, which can support the storage, retrieval and semantic analysis of large-scale vectors; and the vector knowledge base adopts a distributed storage architecture to support efficient reading, writing and retrieval operations of vector data.

[0097] It should be further explained that, in the specific implementation process, the construction process of the vector relationship network includes:

[0098] Semantic similarity is calculated on the obtained vector knowledge base to obtain the semantic similarity between different vectors in the vector knowledge base; and a corresponding similarity matrix is constructed based on the similarity matrix; the matrix elements in the similarity matrix represent the similarity values between corresponding vector pairs; semantic similarity calculation is achieved by combining multiple distance measurement methods to capture semantic relationships from different angles; the distance measurement methods used include using Euclidean distance to calculate the geometric distance between vectors, using cosine similarity to calculate the angular similarity between vectors, using L1 distance to calculate the absolute difference between vectors, and using L1 distance to consider the correlation between features. In addition, block calculation and parallel processing strategies are adopted in the semantic similarity calculation process to process large-scale vector sets;

[0099] Based on the obtained similarity matrix, high-density semantic association regions are identified by applying threshold filtering and density clustering to obtain corresponding semantic association clusters. Threshold filtering refers to vector screening based on a pre-set similarity threshold, retaining only vector pair associations above the similarity threshold. Density clustering uses algorithms such as DBSCAN to identify high-density semantic association regions based on the similarity relationship between vector pairs. High-density semantic association regions refer to groups of vectors with high similarity and close interconnection. Semantic association clusters are vector subsets with clear boundaries. Different vector pairs within the corresponding vector subset have strong semantic associations.

[0100] The obtained semantic association clusters are subjected to relationship type identification and attribute labeling to obtain corresponding semantic relationship triples. Relationship type identification refers to inferring the possible relationship types between vector pairs through rule-based reasoning algorithms and combining the relative position, distance, and distribution characteristics between vectors. For example, "inclusion," "similarity," "precondition," and "causality" are used. Attribute labeling is used to add attribute information such as weight, confidence, and timeliness to the inferred relationship types. Semantic relationship triples are expressed in the form of (head entity, relationship type, tail entity) and can be used to reflect the semantic relationship between vectors.

[0101] Then, a relational graph structure between vectors is constructed based on semantic relationship triples, and graph embedding optimization is performed to obtain the corresponding initial vector relationship network. The relational graph structure is a graph network structure with vectors as nodes and semantic relationships as edges. Edge attribute information includes relationship types, etc. The graph embedding optimization uses knowledge graph embedding methods such as TransE, RotatE, or ComplEx to learn vector representations of semantic relationships, so that the head entity vector is close to the tail entity vector after semantic relationship transformation.

[0102] Perform relational reasoning and link prediction on the initial vector relationship network to supplement potential associations and obtain the corresponding enhanced vector relationship network. Relational reasoning refers to inferring unclear relationships based on existing semantic relationship triples and transitive rules through rule reasoning or statistical reasoning methods. For example, "If A contains B and B contains C, then A may contain C." Link prediction refers to predicting potential associations that may exist but have not yet been labeled in the corresponding initial vector relationship network through graph neural networks or path sorting algorithms. Potential associations are semantic relationships that may exist but are not directly reflected.

[0103] The obtained enhanced vector relational network is subjected to weight optimization and path analysis to obtain the corresponding vector relational network; weight optimization refers to adjusting the weights of edges based on factors such as the importance and credibility of semantic relationships; path analysis is used to identify relational paths and central nodes in the enhanced vector relational network to optimize the network structure; relational paths include shortest paths and critical paths; the vector relational network contains rich semantic relationship types, weights, and path information, providing support for complex semantic queries.

[0104] It should be further explained that, in the specific implementation process, the process of obtaining the query vector representation includes:

[0105] A user interaction unit is provided, and based on the unit, mixed-modal query information input by the target user is obtained; the user interaction unit provides multiple input interfaces, supporting multiple query methods such as text input, voice input, image upload, and video upload; mixed-modal query information is composed of a single modality or a combination of multiple modalities, such as text + image, voice + text, etc.;

[0106] Perform modal separation and recognition on the input mixed-modal query information to distinguish different modal components such as text, image, and audio within the mixed-modal query information. Modal separation refers to separating the input mixed-modal query information into query components of different modalities such as text, image, and audio based on methods such as content feature recognition, MIME type analysis, or feature detection. For example, it can identify the type of text query (keywords, natural language questions, commands, etc.), the type of image query (photos, screenshots, sketches, etc.), and the type of audio query (voice commands, music clips, etc.). The recognition process refers to determining the modal type corresponding to the query component; the corresponding modal-specific query component is obtained through modal classification and recognition.

[0107] Then, the obtained modality-specific query components are normalized to obtain corresponding standardized query representations. Normalization includes operations such as text normalization, image preprocessing, and audio feature extraction. For example, text queries are subjected to text normalization operations such as word segmentation, stemming, and stop word filtering; image queries are subjected to image preprocessing operations such as resizing, normalization, enhancement, and feature extraction; and audio queries are subjected to audio feature extraction operations such as noise reduction, segmentation, and feature transformation. Through the normalization process, query components of different modalities are converted into a standard format.

[0108] The obtained standardized query representation is vectorized and mapped using a multimodal large model, and converted into a vector representation in a unified semantic space to obtain the corresponding query initial vector. The vectorization mapping is implemented based on a pre-built multimodal large model to ensure that the obtained query initial vector and the obtained efficient dense vector are in the same semantic space. The query initial vector is a set of vector representations of query components of each modality. For example, semantic vectors can be extracted for text queries; visual features can be extracted for image queries. The feature vectors of each modality are mapped to the same semantic space to ensure semantic consistency of queries in different modalities.

[0109] Furthermore, the constructed vector relationship network is used to understand and expand the input mixed-modal query information. Intent understanding refers to parsing the potential meaning of the corresponding mixed-modal query information through context understanding and the query initial vector, and performing processes such as ambiguity elimination and implicit information supplementation. Based on the intent understanding results, the original mixed-modal query information is rewritten and expanded to generate supplementary information such as synonyms, related concepts, or hyponyms and hyponyms, and the corresponding intent types (such as information search, recommendation requests, operation instructions, etc.) and focus are extracted. Based on the intent understanding results and expansion results, a more comprehensive query semantic representation is formed to obtain the corresponding intent enhancement description.

[0110] Based on the intent-enhanced description, the obtained initial query vector is reweighted and adjusted to obtain an intent-enhanced query vector. The reweighting process involves adjusting the weights of the query components corresponding to the initial query vector based on the intent-enhanced description and in combination with the attention mechanism to strengthen key semantic features and suppress secondary features. The adjustment process shifts the query vector's position in the semantic space to bring it closer to the vector position corresponding to the user's true intent.

[0111] Contextual analysis is performed on the query vectors with enhanced intent, and combined with the user's historical query behavior to obtain the corresponding query vector representation. Contextual analysis involves identifying potential query sequence relationships by obtaining historical mixed-modal query information in the current session and the semantic similarity between the query results and the current mixed-modal query information. This provides contextual support for the current query. User historical query behavior, including historical query content, click history, and dwell time, is used to infer user interests and preferences. The query vector representation is a comprehensive vector representation that integrates multimodal information, intent understanding, and contextual association.

[0112] It should be further explained that, in the specific implementation process, the process of obtaining the initial search result set includes:

[0113] A multi-level retrieval strategy is designed based on the constructed hierarchical index structure, and a coarse-grained retrieval is performed in combination with the obtained query vector representation to obtain the corresponding candidate result set. The stacking retrieval strategy uses a phased approach of coarse-grained retrieval and fine-grained retrieval to quickly screen the corresponding vector relationship network based on the query vector representation to obtain the corresponding candidate result set. Among them, the coarse-grained retrieval is implemented by using an approximate nearest neighbor algorithm to quickly identify vector sets similar to the query vector expression. The fine-grained retrieval performs exploratory retrieval based on semantic path reasoning based on the corresponding vector set to ensure that potentially relevant results are not missed. The semantic path reasoning process is implemented based on the breadth-first search algorithm.

[0114] The candidate result set is filtered based on pre-set metadata filtering conditions to obtain a streamlined candidate set. Metadata filtering conditions include attribute restrictions such as time range, source restrictions, and modality type to narrow the search scope. The filtering process uses Boolean logic to combine multiple metadata filtering conditions to achieve accurate matching of complex query requirements. The filtering process uses bitmap indexes to achieve efficient filtering.

[0115] A cross-encoder model is used to calculate fine-grained similarity on the reduced candidate set, obtaining an accurate similarity score. The reduced candidate set is then sorted based on this similarity score to obtain a corresponding ordered list of results. The cross-encoder model is an encoder based on the Transformer structure that can simultaneously receive queries and candidates as input, and can be used to capture more complex semantic interactions. The fine-grained similarity calculation process must consider multiple dimensions, including semantic relevance, structural matching, and contextual adaptability.

[0116] Furthermore, based on the ordered result list, the distribution and correlation of data corresponding to different modal types in the retrieval results are obtained, and multimodal fusion and complementary enhancement are performed based on them to obtain an initial retrieval result set with modal balance; the distribution and correlation of data corresponding to different modal types can be used to evaluate the coverage and diversity of content of different modal types; multimodal fusion refers to the integration of retrieval results of different modal types to improve the comprehensiveness and diversity of retrieval results; complementary enhancement refers to the use of the correlation between data corresponding to different modal types to improve the quality and richness of retrieval results; through multimodal fusion and complementary enhancement, the reasonable distribution of retrieval results among different modal types can be ensured to avoid a single modality dominating the final results.

[0117] It should be further explained that, in the specific implementation process, the process of obtaining the optimized search result set includes:

[0118] Feedback the initial search results to the target user, and collect user feedback information and user interaction behavior on the initial search results based on the user interaction unit; explicit feedback includes user clicks, dwell time, collection, rating and other information;

[0119] Based on the feedback information corresponding to the target user, the corresponding initial search results are marked to obtain the search content selected by the user; the implicit association pattern between the initial search results and the input mixed-modal query information is analyzed to obtain the corresponding query-result association graph; the implicit association pattern refers to the potential relationship between the query and the result inferred from the user's interaction behavior, such as the user's tendency to select results of a specific modality or a specific topic; the query-result association graph is a bipartite graph structure that represents the strength of the association between the query and the relevant results;

[0120] Furthermore, the historical feedback information, historical user interaction behaviors, and historical user query behaviors are integrated, cleaned, and standardized to form structured user preference data.

[0121] Based on user preference data and the query-result association graph, a reinforcement learning method is used to adjust the designed multi-level retrieval strategy to form a personalized retrieval model. The corresponding adjustment process includes: modeling the retrieval task formed by the multi-level retrieval strategy as a Markov decision process (MDP), where the state is the current query and result set, the action is the result ranking adjustment, and the reward is the user satisfaction indicator; using a deep Q network (DQN) or policy gradient method to learn the optimal ranking strategy; the learning goal is to maximize cumulative user satisfaction; and then, through reinforcement learning, continuously optimize the ranking strategy to form a personalized retrieval model.

[0122] Furthermore, the initial search results and user preference data are obtained to optimize the threshold parameters; a corresponding adaptive threshold strategy is obtained; the adaptive threshold strategy refers to dynamically adjusting the original similarity threshold based on the distribution and relevance of data corresponding to different modal types in the initial search results, query difficulty, and historical feedback information; the adaptive threshold strategy includes a multi-level structure combining global thresholds and local thresholds to adapt to the needs of different search scenarios; the dynamic adjustment process is implemented through the Bayesian optimization method;

[0123] The personalized retrieval model and adaptive threshold strategy are combined to re-rank and optimize the initial retrieval result set to obtain the corresponding optimized retrieval result set; the re-ranking process is performed by comprehensively considering multiple factors such as semantic similarity between vectors, user preferences, result diversity and timeliness.

[0124] The present invention establishes a standardized multimodal dataset by comprehensively preprocessing and normalizing the collected multi-source heterogeneous modal datasets; uses a pre-trained multimodal large model to perform deep feature extraction and generate knowledge vector representations with unified semantic expression capabilities; constructs a vector knowledge graph through semantic association analysis and hierarchical clustering to form a structured vector knowledge base; finely models the relationships between vectors to build a semantically rich vector relationship network; achieves in-depth understanding and intent analysis of user queries, and obtains initial retrieval results in combination with efficient retrieval strategies; and performs dynamic optimization based on user feedback to improve the relevance of retrieval results and user satisfaction. This method combines the semantic understanding capabilities of the multimodal large model with the efficient characteristics of vector retrieval to build an intelligent, efficient, and sustainably evolving multimodal knowledge retrieval system.

[0125] Example 2

[0126] See also Figure 2 As shown, for parts not described in detail in this embodiment, please refer to the description of Example 1. A dynamic vector knowledge base construction and retrieval system based on a multimodal large model is provided, including:

[0127] The data preprocessing module is used to obtain a multi-source heterogeneous modal dataset, and perform preprocessing and modal normalization on the obtained multi-source heterogeneous modal dataset to obtain a corresponding standardized multimodal dataset;

[0128] The semantic vector generation module extracts features and generates semantic vector representations for standardized multimodal datasets based on a pre-built multimodal large model to obtain the corresponding multimodal knowledge vector set;

[0129] The knowledge base construction module performs semantic association analysis and hierarchical clustering on the obtained multimodal knowledge vector set, constructs a vector knowledge graph structure, and builds a structured vector knowledge base based on the vector knowledge graph structure;

[0130] Network modeling module: used to calculate semantic similarity and model the relationship between vectors on the obtained vector knowledge base to obtain the corresponding vector relationship network;

[0131] The query retrieval module is used to perform multimodal query understanding and intent analysis on the mixed-modal query information input by the user to obtain the corresponding query vector representation; and to perform efficient similarity retrieval in combination with the vector relationship network to obtain the corresponding initial retrieval result set;

[0132] Dynamic optimization module; used to perform dynamic feedback optimization on the initial search result set obtained to obtain an optimized search result set.

[0133] The modules are connected via wired and / or wireless means to achieve data transmission between modules.

[0134] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will be able to modify the technical solutions described in the foregoing embodiments or to substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

[0135] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0136] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0137] In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0138] In the description of the present invention, “several” means one or more, and “a large number” means two or more.

[0139] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0140] The formulas in this manual are all dimensionless and calculated using numerical values. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters and thresholds in the formulas are set by technicians in this field based on actual conditions.

[0141] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A dynamic vector knowledge base construction and retrieval method based on a multimodal large model, characterized by: include: Step 1: Obtain a multi-source heterogeneous modal dataset, and perform preprocessing and modal normalization on the obtained multi-source heterogeneous modal dataset to obtain the corresponding standardized multi-modal dataset; Step 2: Based on the pre-built multimodal large model, feature extraction and semantic vector representation generation are performed on the standardized multimodal dataset to obtain the corresponding multimodal knowledge vector set; Step 3: Perform semantic association analysis and hierarchical clustering on the obtained multimodal knowledge vector set to construct a vector knowledge graph structure, and then build a structured vector knowledge base based on the vector knowledge graph structure; Step 4: Calculate semantic similarity and model the relationship between vectors on the obtained vector knowledge base to obtain the corresponding vector relationship network, including: Perform semantic similarity calculation on the obtained vector knowledge base to obtain the semantic similarity between different vectors in the vector knowledge base; and construct a corresponding similarity matrix based on the semantic similarity; Based on the obtained similarity matrix, high-density semantic association regions are identified by applying threshold filtering and density clustering to obtain corresponding semantic association clusters; relationship types are identified and attributes are labeled on the obtained semantic association clusters to obtain corresponding semantic relationship triples; Constructing a relational graph structure between vectors based on the semantic relation triples, and performing graph embedding optimization to obtain a corresponding initial vector relational network; and performing relational reasoning and link prediction on the initial vector relational network to obtain a corresponding enhanced vector relational network; Perform weight optimization and path analysis on the obtained enhanced vector relationship network to obtain a corresponding vector relationship network; Step 5: Perform multimodal query understanding and intent analysis on the mixed-modal query information input by the user to obtain the corresponding query vector representation; and combine it with the vector relationship network to perform efficient similarity search to obtain the corresponding initial search result set; Step 6: Perform dynamic feedback optimization on the initial search result set to obtain an optimized search result set.

2. The method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model according to claim 1 is characterized in that: A data acquisition unit is provided, which is used to collect and integrate different modal data sets from multiple data sources to obtain corresponding multi-source heterogeneous modal data sets; the multi-source heterogeneous modal data sets include text data, image data, audio data and video data.

3. The method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model according to claim 2 is characterized in that: The process of obtaining a standardized multimodal dataset includes: Perform modality identification and format conversion on the obtained multi-source heterogeneous modal dataset to obtain the corresponding initial modal dataset; Perform data cleaning and outlier detection on the obtained initial modal data set to obtain the corresponding purified modal data set; Performing data enhancement and normalization processing on the obtained purified modal dataset to obtain the corresponding enhanced modal dataset; Perform cross-modal alignment and time synchronization on the obtained enhanced modality dataset to obtain the corresponding multimodal aligned dataset; The obtained multimodal aligned dataset is subjected to feature normalization and dimension consistency processing to obtain the corresponding standardized multimodal dataset.

4. The method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model according to claim 3 is characterized in that: The process of acquiring a multimodal knowledge vector set includes: Based on known modality types, pre-trained large model components suitable for different modal data are selected; the selected pre-trained large model components are integrated and a corresponding multimodal large model is constructed based on them; Based on the constructed multimodal large model, feature extraction is performed on the corresponding standardized multimodal dataset to obtain the corresponding modal feature matrix; Performing cross-modal attention fusion and semantic alignment on the modal feature matrix to obtain a fused feature representation in a unified semantic space; Perform knowledge distillation and feature compression on the obtained fusion feature representation to obtain the corresponding efficient dense vector; The obtained efficient dense vectors are subjected to semantic similarity verification and quality assessment to obtain the corresponding multimodal knowledge vector set.

5. The method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model according to claim 4 is characterized in that: The construction process of the vector knowledge graph structure includes: Perform similarity calculation and nearest neighbor analysis on the obtained multimodal knowledge vector set, and build the corresponding initial semantic association network based on it; Based on the community detection algorithm, the constructed initial semantic association network is subjected to semantic cluster identification to obtain the corresponding semantic community structure; Based on the obtained semantic community structure, a multi-level semantic classification system is constructed by applying a hierarchical clustering algorithm to obtain a corresponding hierarchical semantic tree; the node relationships of the obtained hierarchical semantic tree are enhanced, and cross-level and cross-branch semantic associations are added to obtain a corresponding vector knowledge graph structure.

6. The method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model according to claim 5 is characterized in that: The process of building a vector knowledge base includes: Use a multi-level indexing mechanism to organize efficient dense vectors and design a corresponding hierarchical index structure based on it; the multi-level indexing mechanism can include inverted index and tree index; The obtained efficient dense vector is compressed and encoded to obtain a quantized vector with low storage cost; and a corresponding quantization-restoration conversion model is constructed based on the vector; Based on the vector knowledge graph structure, the hierarchical index structure and quantization-reduction conversion model between vectors are integrated, and a multi-level vector knowledge base is constructed based on it.

7. The method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model according to claim 1 is characterized in that: The process of obtaining the query vector representation includes: Setting a user interaction unit and obtaining mixed-modal query information input by a target user based on the user interaction unit; Perform modal separation and identification on the input mixed-modal query information; obtain corresponding modal-specific query components; normalize the obtained modal-specific query components to obtain corresponding standardized query representations; The obtained standardized query representation is vectorized and mapped using a multimodal large model, and converted into a vector representation in a unified semantic space to obtain the corresponding query initial vector; Based on the constructed vector relationship network, the input mixed-modal query information is understood and expanded to obtain the corresponding enhanced intent description; based on this, the obtained initial query vector is reweighted and adjusted to obtain the enhanced intent query vector; Perform contextual analysis on the query vector with enhanced intent and combine it with the user's historical query behavior to obtain the corresponding query vector representation.

8. The method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model according to claim 7 is characterized in that: The process of obtaining the initial search result set includes: Design a multi-level search strategy based on the constructed hierarchical index structure, and perform a coarse-grained search based on the obtained query vector representation to obtain a corresponding candidate result set; and filter the candidate result set based on pre-set metadata filtering conditions to obtain a streamlined candidate set; the metadata filtering conditions include time range, source restriction, and modality type; Performing fine-grained similarity calculation on the reduced candidate set to obtain an accurate similarity score; and sorting the reduced candidate set based on the similarity score to obtain a corresponding ordered result list; Based on the ordered result list, the distribution and correlation of data corresponding to different modal types in the retrieval results are obtained, and multimodal fusion and complementary enhancement are performed based on them to obtain an initial retrieval result set with modal balance.

9. The method for constructing and retrieving a dynamic vector knowledge base based on a multimodal large model according to claim 8, characterized in that: The process of obtaining the optimized search result set includes: Feedback the obtained initial search results to the target user, and collect the user's feedback information and user interaction behavior on the initial search results based on the user interaction unit; Based on the feedback information corresponding to the target user, the corresponding initial search results are marked to obtain the search content selected by the user; and by analyzing the implicit association pattern between it and the input mixed-modal query information, the corresponding query-result association graph is constructed; Integrate the historical feedback information, historical user interaction behavior, and historical user query behavior obtained; clean and standardize them to form structured user preference data; Based on user preference data and query-result association graph, the designed multi-level retrieval strategy is adjusted using reinforcement learning methods to form a personalized retrieval model; Obtain initial search results and user preference data to optimize threshold parameters; obtain corresponding adaptive threshold strategies; The personalized retrieval model and the adaptive threshold strategy are combined to re-rank and optimize the initial retrieval result set to obtain the corresponding optimized retrieval result set.

Images