Full-text retrieval method for unstructured data based on large model
Through semantic extraction and vector transformation of data retrieval requests, combined with semantic annotation index table matching, the problem of low accuracy in the understanding of search requests in the prior art is solved, and a more accurate and comprehensive search of full-text large-model unstructured data is achieved.
Patent Information
- Application Number
- CN202410076898.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2024-01-15
- Filing Date
- 2024-01-19
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-01-19
AI Technical Summary
The existing unstructured data full-text search methods based on large models have low accuracy when understanding the actual meaning of user search requests, resulting in insufficient accuracy and comprehensiveness of multimodal search.
By semantic extraction of data retrieval requests, context features are obtained, request description tags and semantic ontology are generated, and the request description tags and semantic ontology are converted into data retrieval vectors, and the semantic annotation index table of multimodal data in the database is matched to improve the understanding and matching of search requests. accuracy.
It improves the understanding of the actual meaning of the search request, enhances the accuracy and comprehensiveness of multimodal data retrieval, and provides more personalized and accurate data retrieval results.
Smart Images

Figure CN117951243B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a full-text retrieval method for unstructured data based on a large model. Background Art
[0002] The full-text retrieval method for unstructured data based on large models can be applied to various scenarios that require cross-modal data retrieval and matching. It can effectively utilize the semantic information of data in different modalities to achieve more accurate data retrieval and matching.
[0003] In the existing retrieval method, after receiving the user's retrieval request, the retrieval request is simply parsed and the retrieval keywords are generated to perform full-text retrieval of unstructured data based on a large model. In fact, although the user's retrieval request has the intention to search, due to the subjectivity of the expression, only using the above-mentioned keywords will lead to poor understanding of the actual meaning of the retrieval request, thereby reducing the accuracy and comprehensiveness of multimodal retrieval. Summary of the invention
[0004] The purpose of this application is to propose a full-text retrieval method for unstructured data based on a large model, which is used to solve or alleviate the technical problems existing in the prior art.
[0005] The technical solutions provided by the embodiments of this application are as follows:
[0006] A full-text retrieval method for unstructured data based on a large model, comprising:
[0007] Get data retrieval request;
[0008] Parsing the data retrieval request to determine data retrieval content;
[0009] Converting the data retrieval content into a data retrieval vector;
[0010] Based on the data retrieval vector, a match is performed with a semantic annotation index table of unstructured multimodal data in a database to retrieve multimodal data matching the data retrieval request.
[0011] A large-model-based unstructured data full-text retrieval device, comprising:
[0012] A request acquisition unit, used for acquiring a data retrieval request;
[0013] A request parsing unit, used to parse the data retrieval request to determine the data retrieval content;
[0014] A vectorization unit, used for converting the data retrieval content into a data retrieval vector;
[0015] A matching unit is used to match the data retrieval vector with a semantic annotation index table of unstructured multimodal data in a database to retrieve multimodal data matching the data retrieval request.
[0016] An electronic device comprising:
[0017] one or more processors;
[0018] A computer readable medium configured to store one or more programs,
[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any embodiment of the present application.
[0020] A computer program stores a computer executable program, and when the computer executable program is executed, the method described in any embodiment of the present application is implemented.
[0021] In this application, by performing semantic extraction on the data retrieval request, multiple context features can be obtained, so as to more comprehensively understand the meaning of the retrieval request. Based on each context feature, a request description tag is generated, which can more accurately describe the meaning represented by each context feature, thereby improving the understanding of the actual meaning of the retrieval request. Based on the request description tags corresponding to all context features, the semantic ontology of the data retrieval request is generated, and the meanings of different context features are integrated together, so as to more comprehensively understand the meaning of the entire retrieval request. Then, through data retrieval vector conversion, the data retrieval content is converted into a data retrieval vector, and the different features of multimodal data can be uniformly represented, making the matching more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:
[0023] Figure 1 The present invention is a flowchart of a method for full-text retrieval of unstructured data based on a large model according to an embodiment of the present application.
[0024] Figure 2 This is a structural schematic diagram of a large model-based full-text retrieval device for unstructured data according to an embodiment of the present application.
[0025] Figure 3 Schematic diagram of the structure of the electronic device of this embodiment.
[0026] Figure 4Detailed description of the hardware structure of the electronic device of this embodiment. DETAILED DESCRIPTION
[0027] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.
[0028] Figure 1 This is a flow chart of a method for full-text retrieval of unstructured data based on a large model according to an embodiment of the present application. Figure 1 As shown, the following steps are performed based on the large model, which include:
[0029] S101, obtaining a data retrieval request;
[0030] S102, parsing the data retrieval request to determine data retrieval content;
[0031] S103, converting the data retrieval content into a data retrieval vector;
[0032] S104: Based on the data retrieval vector, matching is performed with a semantic annotation index table of unstructured multimodal data in a database to retrieve multimodal data matching the data retrieval request.
[0033] In this embodiment, the data retrieval request in step S101 may be a data retrieval request based on SQL or REST API callable. The format of the data retrieval request may be, for example, JSON, XML, or other formats. For example, if the REST API is callable, an HTTP client (such as curl) or an HTTP library in a programming language may be used to obtain the data retrieval request.
[0034] In the above embodiment, the big model may be, for example, a generative big model or a trained neural network.
[0035] Optionally, parsing the data retrieval request to determine data retrieval content includes:
[0036] Performing semantic extraction on the data retrieval request to obtain multiple context features therein;
[0037] Based on each context feature, a request description tag is generated;
[0038] Generate a semantic ontology of the data retrieval request based on request description tags corresponding to all context features;
[0039] The semantic ontology is used as the data retrieval content.
[0040] The following provides an exemplary code for implementing the above-mentioned parsing of the data retrieval request to determine the data retrieval content:
[0041] # Import library
[0042] import nltk
[0043] from nltk.tokenize import word_tokenize
[0044] from nltk.tag import pos_tag
[0045] from nltk.corpus import wordnet
[0046] from nltk.stem import WordNetLemmatizer
[0047] # Define a function for semantic extraction
[0048] def semantic_extraction(text):
[0049] tokens = word_tokenize(text) # Segment the text
[0050] tags = pos_tag(tokens) # Perform part-of-speech tagging on the text after word segmentation
[0051] lemmatizer = WordNetLemmatizer()
[0052] # Convert part-of-speech tags to WordNet part-of-speech tags
[0053] def get_wordnet_pos(tag):
[0054] if tag.startswith('J'):
[0055] return wordnet.ADJ
[0056] elif tag.startswith('V'):
[0057] return wordnet.VERB
[0058] elif tag.startswith('N'):
[0059] return wordnet.NOUN
[0060] elif tag.startswith('R'):
[0061] return wordnet.ADV
[0062] else:
[0063] return wordnet.NOUN
[0064] # Merge each word into its own word
[0065] lemmatized_tokens = [lemmatizer.lemmatize(word, get_wordnet_pos(tag)) for word, tag in tags]
[0066] return lemmatized_tokens
[0067] # Define a function to generate a request description tag
[0068] def generate_request_labels(tokens):
[0069] labels = []
[0070] for token in tokens:
[0071] # Here, you can define the corresponding request description tag generation rules according to the specific application scenario
[0072] # This is just an example, simply using part of speech as the label
[0073] label = pos_tag([token])[0][1]
[0074] labels.append(label)
[0075] return labels
[0076] # Define a function to generate the semantic ontology of data retrieval request
[0077] def generate_semantic_ontology(labels):
[0078] # Here, you can convert the request description tag into a semantic ontology according to the specific application scenario
[0079] # This is just an example. Simply concatenate the request description tags as the semantic ontology
[0080] ontology = ' '.join(labels)
[0081] return ontology
[0082] # Example data retrieval request
[0083] data_retrieval_request = "Find images of cute cats"
[0084] # Perform semantic extraction
[0085] tokens = semantic_extraction(data_retrieval_request)
[0086] # Generate request description tag
[0087] labels = generate_request_labels(tokens)
[0088] # Generate semantic ontology for data retrieval request
[0089] semantic_ontology = generate_semantic_ontology(labels)
[0090] # Use semantic ontology as data retrieval content
[0091] data_retrieval_content = semantic_ontology
[0092] print("Data retrieval content:", data_retrieval_content)
[0093] In this application, by performing semantic extraction on the data retrieval request, multiple context features can be obtained, so as to more comprehensively understand the meaning of the retrieval request. Based on each context feature, a request description tag is generated, which can more accurately describe the meaning represented by each context feature, thereby improving the understanding of the actual meaning of the retrieval request. Based on the request description tags corresponding to all context features, the semantic ontology of the data retrieval request is generated, and the meanings of different context features are integrated together, so as to more comprehensively understand the meaning of the entire retrieval request. Then, through data retrieval vector conversion, the data retrieval content is converted into a data retrieval vector, and the different features of multimodal data can be uniformly represented, making the matching more accurate.
[0094] Optionally, the performing semantic extraction on the data retrieval request to obtain a plurality of context features therein includes:
[0095] Performing semantic extraction on the data retrieval request to obtain semantic entities and neighbor link relationships between semantic entities;
[0096] Based on the hash mapping between the semantic entities and the neighbor link relationships between the semantic entities, it is determined that the data retrieval request is to be subjected to semantic extraction to obtain a plurality of context features therein.
[0097] To this end, semantic extraction can help the system understand the semantic content of data retrieval requests more comprehensively, identify the entities involved in the request and the relationships between them, and thus more accurately understand the user's intention. By obtaining semantic entities and the relationships between them, data retrieval can be performed more accurately, thereby improving the accuracy and relevance of retrieval results. By analyzing semantic entities and relationships, user needs can be better understood, thereby achieving personalized data recommendations and search result ranking. Using semantic entities and relationships as contextual features can provide more information for search engines and recommendation systems, thereby achieving intelligent search and recommendation functions.
[0098] The following is a code to implement context feature extraction:
[0099] # Import library
[0100] from py2neo import Graph
[0101] # Connect to the graph database
[0102] graph = Graph("bolt: / / localhost:7687", auth=("neo4j", "password"))
[0103] # Define a function for semantic extraction to obtain semantic entities and neighbor link relationships between semantic entities
[0104] def semantic_extraction_with_neo4j(text):
[0105] # Query text-related entities and their neighbor relationships in the graph database
[0106] query = """
[0107] MATCH (n:Entity)-[r]->(m:Entity)
[0108] WHERE n.name in $text
[0109] RETURN n, r, m
[0110] """
[0111] result = graph.run(query, text=text).data()
[0112] # Extract entities and relations
[0113] entities = []
[0114] relationships = []
[0115] for record in result:
[0116] entities.append(record['n']['name'])
[0117] entities.append(record['m']['name'])
[0118] relationships.append(record['r'].type)
[0119] return entities, relationships
[0120] # Define a function to determine the data retrieval request for semantic extraction to obtain multiple context features based on the hash mapping between semantic entities and neighbor link relationships
[0121] def determine_context_features(entities, relationships):
[0122] # Here, you can determine context features based on entities and relationships according to specific application scenarios.
[0123] # This is just an example, simply concatenating entities and relationships as context features
[0124] context_features = entities + relationships
[0125] return context_features
[0126] # Example data retrieval request
[0127] data_retrieval_request = "Find images of cute cats"
[0128] # Perform semantic extraction to obtain semantic entities and neighbor link relationships between semantic entities
[0129] entities, relationships = semantic_extraction_with_neo4j(data_retrieval_request)
[0130] # Determine context features based on hash map
[0131] context_features = determine_context_features(entities,relationships)
[0132] print("Context features:", context_features)
[0133] Optionally, generating a request description tag based on each context feature includes:
[0134] Encode each context feature to obtain an encoding vector;
[0135] Analyze the attention weights between each context feature and other context features to obtain an attention weight vector;
[0136] The encoding vector and the attention weight vector are fused to obtain a request description label corresponding to each context feature.
[0137] In this embodiment, by encoding each context feature, complex features can be converted into a vector form that is easy to process, thereby improving the efficiency and accuracy of subsequent processing. Furthermore, by analyzing the attention weight between each context feature and other context features, the correlation and influence between different features can be captured, which helps to better understand the context information in the data retrieval request. By fusing the encoding vector and the attention weight vector, the interactive information between different features can be fully mined, and the accuracy of the request description label can be improved, thereby improving the accuracy of the understanding of the retrieval request.
[0138] The following is a sample code for generating a request description tag:
[0139] import torch
[0140] from transformers import ModelTokenizer, ModelModel
[0141] # Load the pre-trained label model and tokenizer
[0142] tokenizer = ModelTokenizer.from_pretrained('Model-base-uncased')
[0143] model = ModelModel.from_pretrained('Model-base-uncased')
[0144] # Define the encoding function, use Model to encode each context feature, and get the encoding vector
[0145] def encode_context_features(context_features):
[0146] encoded_vectors = []
[0147] for feature in context_features:
[0148] inputs = tokenizer(feature, return_tensors="pt", padding=True, truncation=True)
[0149] outputs = model(**inputs)
[0150] encoded_vector = outputs.last_hidden_state.mean(dim=1).squeeze()
[0151] encoded_vectors.append(encoded_vector)
[0152] return encoded_vectors
[0153] # Define the attention weight calculation function, analyze the attention weight between each context feature and other features, and get the attention weight vector
[0154] def calculate_attention_weights(encoded_vectors):
[0155] attention_weights = []
[0156] for i in range(len(encoded_vectors)):
[0157] attention_weight = []
[0158] for j in range(len(encoded_vectors)):
[0159] # Calculate the attention weight, which can be calculated based on the specific attention mechanism
[0160] # This is just an example, using simple dot product attention
[0161] attention_weight.append(torch.dot(encoded_vectors[i],encoded_vectors[j]))
[0162] attention_weights.append(attention_weight)
[0163] return attention_weights
[0164] # Define the vector fusion function to fuse the encoding vector and the attention weight vector to obtain the request description label
[0165] def merge_vectors(encoded_vectors, attention_weights):
[0166] merged_vectors = []
[0167] for i in range(len(encoded_vectors)):
[0168] merged_vector = encoded_vectors[i]* attention_weights[i]
[0169] merged_vectors.append(merged_vector)
[0170] return merged_vectors
[0171] # Example context features
[0172] context_features = ["cat", "image", "cute", "animal"]
[0173] # Encode each context feature to obtain the encoding vector
[0174] encoded_vectors = encode_context_features(context_features)
[0175] # Analyze the attention weights between each context feature and other features to get the attention weight vector
[0176] attention_weights = calculate_attention_weights(encoded_vectors)
[0177] #Merge the encoding vector and the attention weight vector to get the request description label
[0178] request_description_labels = merge_vectors(encoded_vectors,attention_weights)
[0179] print("Request description labels:", request_description_labels)
[0180] In this embodiment, the generation of the semantic ontology of the data retrieval request based on the request description tags corresponding to all context features includes:
[0181] Build a knowledge graph based on the request description tags corresponding to all context features;
[0182] Semantic reasoning is performed based on the knowledge graph to generate a semantic ontology of the data retrieval request.
[0183] To this end, the request description tags corresponding to multiple contextual features are comprehensively considered, and the associations and semantic relationships between different tags are captured by constructing a knowledge graph, so as to more comprehensively understand the semantics of data retrieval requests. By constructing a knowledge graph, the solution can fuse and infer the semantic relationships between different request description tags, so as to more accurately understand the user's intentions and needs, and improve the accuracy and depth of semantic understanding. Semantic reasoning based on the knowledge graph can help the system discover the semantic relationships and rules hidden between request description tags, so as to better understand the user's needs and provide users with more accurate data retrieval results. By constructing a semantic ontology, the system can better understand the user's query intentions, thereby improving the intelligence level of the data retrieval system and providing users with more personalized and accurate data retrieval services.
[0184] An exemplary code for generating a semantic ontology of the data retrieval request is provided as follows:
[0185] import networkx as nx
[0186] import pyke
[0187] # Build a knowledge graph based on the request description tags corresponding to all context features
[0188] def build_knowledge_graph(request_description_labels):
[0189] # Create an empty directed graph
[0190] graph = nx.DiGraph()
[0191] # Add request description labels to the graph
[0192] for label in request_description_labels:
[0193] graph.add_node(label)
[0194] # Add the relationship edge between labels. Here, it is assumed that there is a relationship between each label.
[0195] for i in range(len(request_description_labels)):
[0196] for j in range(i+1, len(request_description_labels)):
[0197] graph.add_edge(request_description_labels[i], request_description_labels[j])
[0198] Return graph
[0199] # Perform semantic reasoning based on the knowledge graph to generate the semantic ontology of the data retrieval request
[0200] def semantic_reasoning(knowledge_graph):
[0201] # Use the knowledge graph reasoning engine for semantic reasoning
[0202] engine = pyke.KnowledgeEngine()
[0203] engine.activate('semantic_rules.krb') # Load semantic reasoning rules
[0204] # Perform semantic reasoning to obtain semantic ontology
[0205] semantic_results = engine.prove_1(knowledge_graph, 'semantic_inference')
[0206] return semantic_results
[0207] # Request description tag corresponding to the example context feature
[0208] request_description_labels = ["cat", "image", "cute", "animal"]
[0209] # Build a knowledge graph
[0210] knowledge_graph = build_knowledge_graph(request_description_labels)
[0211] # Perform semantic reasoning based on the knowledge graph to generate the semantic ontology of data retrieval requests
[0212] semantic_results = semantic_reasoning(knowledge_graph)
[0213] print("Semantic ontology:", semantic_results)
[0214] In this embodiment, the construction of the knowledge graph based on the request description tags corresponding to all context features includes:
[0215] Mapping each context feature to an entity in the knowledge graph using each context feature as a key in a key-value pair and a request description tag corresponding to each context feature as a value in the key-value pair;
[0216] The semantic relationship strength between context features is mapped to the link strength between entities in the knowledge graph.
[0217] To this end, by constructing a knowledge graph, each context feature and the corresponding request description label can be mapped to an entity in the knowledge graph, thereby enriching the expression of semantic information, so as to better understand the user's request and provide more accurate data retrieval services. Furthermore, mapping the semantic relationship strength between context features to the link strength between entities in the knowledge graph can better model and express the association and mutual influence between different context features, which helps the system to more comprehensively understand the semantics of user requests and improve the accuracy of data retrieval. In addition, since the knowledge graph is a flexible data structure, it can be easily expanded and updated. When new context features and request description labels appear, they can be easily added to the knowledge graph without large-scale changes to the system. Furthermore, by performing semantic reasoning on the knowledge graph, the system can perform intelligent inference based on the entities and associations in the graph, so as to better understand user requests, provide personalized and accurate data retrieval services, and improve user experience.
[0218] In this embodiment, the semantic reasoning based on the knowledge graph to generate the semantic ontology of the data retrieval request includes:
[0219] Mapping the knowledge graph into the created vector space to obtain a semantic vector table;
[0220] Recursive logical reasoning is performed on the semantic vector table to generate a semantic ontology of the data retrieval request.
[0221] To this end, mapping the knowledge graph into the vector space can represent the entities and relationships in the knowledge graph as vectors, thereby better expressing semantic information in the vector space, facilitating the calculation and comparison of semantic information, and improving the processing efficiency of semantic information. In addition, recursive logical reasoning on the semantic vector table can help the system understand semantic information more deeply, discover hidden semantic relationships, infer more complex semantic structures, improve the intelligence of the system, enable the system to understand user requests more comprehensively, and provide more accurate data retrieval services. Furthermore, mapping the entities and relationships in the knowledge graph into the vector space can achieve a unified semantic representation, so that the semantic relationships between different entities can be calculated and compared in the vector space, thereby better understanding the association and mutual influence between different entities and improving the accuracy of data retrieval. Recursive logical reasoning can help the system discover complex semantic relationships and rules, so that the system has more diverse reasoning capabilities, can better respond to complex user requests and data retrieval needs, and improve the adaptability and flexibility of the system.
[0222] The following is an exemplary code for generating the semantic ontology of the data retrieval request:
[0223] # Import required libraries
[0224] import spacy # for natural language processing
[0225] import numpy as np # for vector representation
[0226] import networkx as nx # used to build knowledge graph
[0227] # Create a knowledge graph
[0228] G = nx.DiGraph()
[0229] G.add_edge('people', 'learning')
[0230] G.add_edge('learning', 'knowledge')
[0231] G.add_edge('knowledge', 'growth')
[0232] # Load the mapping model using spacy
[0233] nlp = spacy.load("en_core_web_sm")
[0234] # Map entities and relationships in the knowledge graph to the vector space
[0235] vectors = {}
[0236] for node in G.nodes:
[0237] vectors[node] = nlp(node).vector
[0238] # Recursive logical reasoning
[0239] def recursive_logic_inference(node, vectors, graph):
[0240] if len(list(graph.successors(node))) == 0:
[0241] return vectors[node]
[0242] else:
[0243] children_vectors = [recursive_logic_inference(child, vectors,graph) for child ingraph.successors(node)]
[0244] # Perform processing for recursive logical reasoning here, simplified to taking the average here
[0245] result_vector = np.mean(children_vectors, axis=0)
[0246] return result_vector
[0247] # Generate the semantic ontology for the data retrieval request
[0248] semantic_entity = recursive_logic_inference('person', vectors, G)
[0249] print(semantic_entity)
[0250] In this embodiment, the conversion of the data retrieval content into a data retrieval vector includes:
[0251] Dividing the data retrieval content into blocks to obtain a plurality of content blocks;
[0252] Perform hidden encoding on each content block to obtain the corresponding hidden representation vector;
[0253] The hidden representation vectors corresponding to all content blocks are aligned, and the aligned hidden representation vectors are weighted merged to obtain the data retrieval vector.
[0254] In this embodiment, by dividing the data retrieval content into blocks and performing hidden coding, a multi-level semantic representation of the content can be achieved, so that complex query content can be better understood and processed, and the ability to understand and express the query content can be improved. By performing hidden coding on each content block, the content block can be mapped to a latent semantic space, thereby achieving a semantic representation of the content, better understanding the semantic relationship between the content, and improving the accuracy and effect of data retrieval. Then, by aligning and weighted merging the hidden representation vectors corresponding to all content blocks, the semantic information of different content blocks can be effectively integrated, thereby obtaining a more comprehensive and accurate data retrieval vector, so that the semantics of the overall query content can be better understood, and the accuracy and efficiency of data retrieval can be improved.
[0255] The following is an exemplary code for converting the data retrieval content into a data retrieval vector:
[0256] import spacy
[0257] import numpy as np
[0258] nlp = spacy.load("en_core_web_sm")
[0259] # Divide the data retrieval content into several content blocks
[0260] def chunk_text(text):
[0261] doc = nlp(text)
[0262] chunks = [chunk.text for chunk in doc.noun_chunks]
[0263] return chunks
[0264] # Perform hidden encoding on each content block to obtain the corresponding hidden representation vector
[0265] def encode_chunk(chunk):
[0266] return nlp(chunk).vector
[0267] # Align the hidden representation vectors corresponding to all content blocks, and weight the aligned hidden representation vectors to obtain the data retrieval vector
[0268] def align_and_merge_vectors(vectors):
[0269] aligned_vectors = vectors # This is simplified to use the original vector directly
[0270] merged_vector = np.mean(aligned_vectors, axis=0) # Take the average value as the merged vector
[0271] return merged_vector
[0272] # Data retrieval content
[0273] search_content = "Apple Inc. is an American multinational technology company that designs, develops, and sells consumer electronics, computer software, and online services."
[0274] # Divide the data retrieval content into chunks
[0275] chunks = chunk_text(search_content)
[0276] #Hidden encoding for each content block
[0277] chunk_vectors = [encode_chunk(chunk) for chunk in chunks]
[0278] # Align the hidden representation vectors corresponding to all content blocks, and weight the aligned hidden representation vectors to obtain the data retrieval vector
[0279] search_vector = align_and_merge_vectors(chunk_vectors)
[0280] print(search_vector)
[0281] In this embodiment, the step of performing hidden coding on each content block to obtain a corresponding hidden representation vector includes:
[0282] Adding a segmentation identifier at the beginning and end of each content block to form a corresponding encoding entity;
[0283] Using the trained convolution kernel, feature extraction is performed on the encoding entity corresponding to each content block to obtain the corresponding hidden representation vector.
[0284] To this end, adding segmentation identifiers at the beginning and end of each content block helps the model better understand the boundaries and structure of the content block, thereby reducing information overlap and confusion, and improving the understanding and encoding of the content block. Furthermore, by using the trained convolution kernel to extract features from the encoding entity corresponding to each content block, the local features and semantic information in the content block can be effectively captured, thereby achieving more accurate and rich content block encoding and improving the semantic understanding and representation capabilities of the content block.
[0285] The following provides an exemplary code for performing hidden encoding on each content block to obtain a corresponding hidden representation vector:
[0286] from transformers import Model1Tokenizer, Model1Model
[0287] import torch
[0288] # Load the pre-trained Model1 model and tokenizer
[0289] tokenizer = Model1Tokenizer.from_pretrained('Model1-base-uncased')
[0290] model = Model1Model.from_pretrained('Model1-base-uncased')
[0291] #Hiddenly encode each content block and add a segmentation identifier
[0292] def encode_chunk_with_separator(chunk):
[0293] encoded_chunk = tokenizer.encode(chunk, add_special_tokens=True,return_tensors='pt')
[0294] return encoded_chunk
[0295] # Use the pre-trained Model1 model to extract features from the encoding entity corresponding to each content block to obtain the corresponding hidden representation vector
[0296] def extract_features(encoded_chunk):
[0297] with torch.no_grad():
[0298] outputs = model(encoded_chunk)
[0299] hidden_states = outputs[0]
[0300] # Take the hidden state of the last layer as the representation vector
[0301] representation = hidden_states[-1].mean(dim=1) # Take the average of the representation vector of each token
[0302] return representation
[0303] # Data retrieval content
[0304] search_content = "Apple Inc. is an American multinational technology company that designs, develops, and sells consumer electronics, computer software, and online services."
[0305] # Divide the data retrieval content into chunks
[0306] chunks = ["Apple Inc.", "an American multinational technologycompany", "consumer electronics", "computersoftware", "online services"]
[0307] #Hiddenly encode each content block and add a segmentation identifier
[0308] encoded_chunks = [encode_chunk_with_separator(chunk) for chunk inchunks]
[0309] # Use the pre-trained Model1 model to extract features from the encoding entity corresponding to each content block to obtain the corresponding hidden representation vector
[0310] chunk_vectors = [extract_features(encoded_chunk) for encoded_chunk inencoded_chunks]
[0311] # Output the hidden representation vector corresponding to each content block
[0312] for i, vector in enumerate(chunk_vectors):
[0313] print(f"Hidden representation for chunk {i+1}: {vector}")
[0314] In this embodiment, aligning the hidden representation vectors corresponding to all content blocks includes:
[0315] The bits corresponding to the segmentation identifier of each content block in the hidden representation vector are used as boundaries, and the hidden representation vectors corresponding to all the content blocks are normalized to align the hidden representation vectors corresponding to all the content blocks.
[0316] To this end, by aligning the hidden representation vectors corresponding to all content blocks with the bits corresponding to the segmentation identifier of each content block in the hidden representation vector as the boundary, it can be ensured that the representation vectors of each content block have the same length, so as to facilitate unified operations in subsequent processing, such as calculating similarity or performing other vector operations. In addition, normalizing the hidden representation vectors corresponding to all content blocks can make them have a unified scale, so that they are easier to compare and analyze in subsequent calculations. Normalization can eliminate the influence of vector length, making the similarity calculation between vectors more accurate. By aligning and normalizing the hidden representation vectors, the generalization ability can be improved, thereby reducing the interference caused by different lengths and scales, and being more robust when processing inputs of different lengths and scales.
[0317] The following is an exemplary code for aligning the hidden representation vectors corresponding to all content blocks:
[0318] import torch
[0319] # Align the hidden representation vectors corresponding to all content blocks
[0320] def align_vectors(chunk_vectors):
[0321] # Find the position corresponding to the split identifier
[0322] special_token_positions = []
[0323] for vector in chunk_vectors:
[0324] special_token_positions.append(torch.where(vector[0]==tokenizer.sep_token_id)[0].item())
[0325] # Find the largest split identifier position
[0326] max_position = max(special_token_positions)
[0327] # Align each vector
[0328] aligned_vectors = []
[0329] for vector, position in zip(chunk_vectors, special_token_positions):
[0330] if position <max_position:
[0331] aligned_vector = torch.cat((vector, torch.zeros((1, max_position - position,vector.shape[-1]))), dim=1)
[0332] else:
[0333] aligned_vector = vector
[0334] aligned_vectors.append(aligned_vector)
[0335] return aligned_vectors
[0336] # Normalize the hidden representation vectors corresponding to all content blocks
[0337] def normalize_vectors(chunk_vectors):
[0338] normalized_vectors = [torch.nn.functional.normalize(vector, p=2,dim=-1) for vector inchunk_vectors]
[0339] return normalized_vectors
[0340] # Divide the data retrieval content into chunks
[0341] chunks = ["Apple Inc.", "an American multinational technologycompany", "consumer electronics", "computersoftware", "online services"]
[0342] # Example hidden representation vector
[0343] chunk_vectors = [torch.randn(1, 10, 768), torch.randn(1, 12, 768), torch.randn(1, 8, 768), torch.randn(1, 10, 768), torch.randn(1, 11, 768)]
[0344] # Align the hidden representation vectors corresponding to all content blocks
[0345] aligned_vectors = align_vectors(chunk_vectors)
[0346] # Normalize the hidden representation vectors corresponding to all content blocks
[0347] normalized_vectors = normalize_vectors(aligned_vectors)
[0348] # Output aligned and normalized hidden representation vector
[0349] for i, vector in enumerate(normalized_vectors):
[0350] print(f"Normalized vector for chunk {i+1}: {vector}")
[0351] In the above code implementation, a function align_vectors is first defined to align the hidden representation vectors corresponding to all content blocks. In this function, the position corresponding to the segmentation identifier in each hidden representation vector is found, and the position of the largest segmentation identifier is found. Then each vector is aligned so that they have the same length. Next, a function normalize_vectors is defined to normalize the hidden representation vectors corresponding to all content blocks. In this function, PyTorch's normalize function is used to normalize each vector using the L2 norm. Finally, the example hidden representation vectors are aligned and normalized, and the processed results are output.
[0352] In this embodiment, weighted merging of the aligned hidden representation vectors to obtain a data retrieval vector includes:
[0353] Perform semantic strength analysis on the aligned hidden representation vectors to assign vector weights according to the magnitude of semantic strength;
[0354] Based on the vector weights, the aligned hidden representation vectors are weighted and merged to obtain a data retrieval vector.
[0355] To this end, by performing semantic strength analysis on the aligned hidden representation vectors, vector weights can be assigned according to the semantic importance and contribution of each content block, thereby more accurately capturing the importance of each content block, and better reflecting the semantic information of the content block during the merging process. Furthermore, based on the vector weights, weighted merging of the aligned hidden representation vectors can effectively integrate the semantic information of each content block into the final data retrieval vector, ensuring that the contribution of important content blocks is fully reflected when performing retrieval, thereby improving the accuracy and effect of retrieval. In addition, by performing semantic strength analysis and weighted merging on the aligned hidden representation vectors, the semantic information of different content blocks in the text can be better captured, thereby improving the effect and accuracy of data retrieval, and better utilizing the information in the hidden representation vectors, so that the retrieval vector more accurately represents the semantic content of the retrieval request.
[0356] The following is an exemplary code to implement weighted merging of the aligned hidden representation vectors to obtain a data retrieval vector:
[0357] import numpy as np
[0358] # Assume that we already have the aligned hidden representation vector
[0359] aligned_hidden_rep_vectors = np.array([[0.1, 0.2, 0.3], [0.4, 0.5,0.6], [0.7, 0.8, 0.9]])
[0360] # Semantic strength analysis, assigning vector weights according to the semantic strength
[0361] def analyze_semantic_intensity(aligned_hidden_rep_vectors):
[0362] vector_weights = {}
[0363] for i, vector in enumerate(aligned_hidden_rep_vectors):
[0364] semantic_intensity = np.linalg.norm(vector) # The semantic intensity calculation method can be adjusted according to the actual situation
[0365] vector_weights[i] =semantic_intensity
[0366] return vector_weights
[0367] vector_weights = analyze_semantic_intensity(aligned_hidden_rep_vectors)
[0368] # Based on the vector weight, weighted merge the hidden representation vector to obtain the data retrieval vector
[0369] def weighted_merge(aligned_hidden_rep_vectors, vector_weights):
[0370] weighted_vector = np.zeros(aligned_hidden_rep_vectors.shape[1])
[0371] for i, weight in vector_weights.items():
[0372] weighted_vector += aligned_hidden_rep_vectors[i]* weight
[0373] return weighted_vector
[0374] data_retrieval_vector = weighted_merge(aligned_hidden_rep_vectors,vector_weights)
[0375] # Output data retrieval vector results
[0376] print(data_retrieval_vector)
[0377] In this embodiment, matching the data retrieval vector with a semantic annotation index table of unstructured multimodal data in a database to retrieve multimodal data matching the data retrieval request includes:
[0378] The semantic similarity between the data retrieval vector and the semantic annotation index table of unstructured multimodal data in the database is calculated to retrieve multimodal data matching the data retrieval request.
[0379] To this end, by calculating the semantic similarity between the data retrieval vector and the multimodal data, the system can more accurately find data that matches the user's query intent, avoiding the inaccurate results that may be caused by simple keyword matching, thereby improving the accuracy of the retrieval. In addition, since the data retrieval vector contains the deep semantic information of the user's query, by matching and calculating the similarity of this information, the system can more comprehensively understand the user's actual query intent, and facilitate the provision of results that better meet the user's needs. In addition, through the calculation of semantic similarity, it can flexibly match multimodal data of different types and forms, adapt to various complex and diverse data structures, and improve its adaptability, so that it can be easily extended to large-scale multimodal data sets. By improving computational efficiency and optimizing algorithms, it can process large-scale data and maintain efficient retrieval performance. Furthermore, through the calculation of semantic similarity, users are provided with the explainability of data matching, so that they can better understand why certain data matches their queries, thereby enhancing users' trust and satisfaction with the retrieval results.
[0380] Optionally, the semantic annotation index table of the multimodal data includes each type of multimodal data and a corresponding semantic annotation index vector:
[0381] Calculating the semantic similarity between the data retrieval vector and the semantic annotation index table of unstructured multimodal data in the database, including:
[0382] The data retrieval vector and the semantic annotation index vector are bitwise ANDed to calculate the semantic similarity between the data retrieval vector and the semantic annotation index table of unstructured multimodal data in the database.
[0383] To this end, a fast and low-complexity operation is achieved through the bitwise AND operation, which improves the efficiency of similarity calculation and reduces computing time and resource consumption. In addition, through the bitwise AND operation, it is possible to make full use of each bit of information in the data, capture the correlation and dependency between features, and help to more accurately calculate semantic similarity and improve the accuracy of retrieval. Furthermore, due to the simplicity and efficiency of the bitwise AND operation, the scheme can be easily extended to larger data sets and more complex multimodal data structures, and can handle large-scale and diverse multimodal data to meet the growing data needs. Furthermore, the bitwise AND operation has a certain robustness to noise and outliers. Even in the presence of certain noise or outliers, it can still calculate semantic similarity relatively accurately and provide reliable data retrieval results.
[0384] In this embodiment, the semantic annotation index table of the multimodal data includes a modality dictionary, and the modality dictionary includes a modality bitmap index and a corresponding semantic annotation index vector:
[0385] Calculating the semantic similarity between the data retrieval vector and the semantic annotation index table of unstructured multimodal data in the database includes:
[0386] The data retrieval vector is matched with the modal bitmap index, and a bitwise AND operation is performed with the semantic annotation index vector corresponding to the matched modal bitmap index to calculate the semantic similarity between the data retrieval vector and the semantic annotation index table of unstructured multimodal data in the database.
[0387] To this end, by using the modal bitmap index, the system can quickly locate the modal bitmap index that matches the data retrieval vector, thereby reducing the time and resource consumption of similarity calculation and improving retrieval efficiency. In addition, the modal bitmap index and the corresponding semantic annotation index vector together constitute the semantic annotation index table of multimodal data. By matching the data retrieval vector with the modal bitmap index and calculating the bitwise AND result of the semantic annotation index vector corresponding to the matching modal bitmap index, the semantic information of the data can be understood more deeply and the semantic similarity can be calculated more accurately. Furthermore, the use of the modal bitmap index can flexibly expand the semantic annotation index table of multimodal data. By adding or deleting the modal bitmap index and the corresponding semantic annotation index vector, the system can adapt to multimodal data of different types and sizes and meet the ever-changing data processing needs. By matching with the modal bitmap index, the modal bitmap index related to the data retrieval vector can be accurately found, the accuracy of the similarity calculation can be improved, and more reliable data retrieval results can be provided.
[0388] The following is an exemplary code for calculating the semantic similarity between the data retrieval vector and the semantic annotation index table of unstructured multimodal data in the database:
[0389] import numpy as np
[0390] data_retrieval_vector = np.array([1, 0, 1, 0, 1])
[0391] modality_dictionary = {
[0392] "Modal 1": {
[0393] "Bitmap index": np.array([1, 0, 0, 1, 0]),
[0394] "Semantic annotation index vector": np.array([1, 0, 1, 0, 1])
[0395] },
[0396] "Modal 2": {
[0397] "Bitmap index": np.array([0, 1, 1, 0, 0]),
[0398] "Semantic annotation index vector": np.array([0, 1, 0, 1, 1])
[0399] },
[0400] "Modal 3": {
[0401] "Bitmap index": np.array([0, 0, 1, 1, 1]),
[0402] "Semantic annotation index vector": np.array([1, 1, 0, 0, 0])
[0403] }
[0404] }
[0405] # Match the data retrieval vector with the modal bitmap index
[0406] matched_modality = None
[0407] for modality, modality_info in modality_dictionary.items():
[0408] if np.all(data_retrieval_vector == modality_info["bitmap index"]):
[0409] matched_modality = modality
[0410] break
[0411] # If a matching modality is found, calculate the bitwise AND result of the semantic annotation index vector corresponding to the matching modality bitmap index
[0412] if matched_modality is not None:
[0413] semantic_annotation_index_vector = modality_dictionary[matched_modality]["Semantic Annotation Index Vector"]
[0414] bitwise_and_result = np.bitwise_and(data_retrieval_vector,semantic_annotation_index_vector)
[0415] similarity = np.sum(bitwise_and_result)
[0416] print(f"The modality that matches the data retrieval vector is: {matched_modality}")
[0417] print(f"Semantic similarity is: {similarity}")
[0418] else:
[0419] print("No matching mode found for data retrieval vector")
[0420] In this embodiment, the data retrieval vector is matched with the modality bitmap index, including:
[0421] The data retrieval vector is dot-producted with the binary modality identifier in the modality bitmap index to determine whether the data retrieval vector matches the modality bitmap based on the size of the dot product.
[0422] To this end, by calculating the dot product of the data retrieval vector and the binary modal identifier in the modal bitmap index, the system can quickly determine whether the data retrieval vector matches the modal bitmap, thereby improving the matching efficiency. In addition, the result of the dot product can intuitively reflect the similarity between the data retrieval vector and the modal bitmap. When the dot product result is close to 0, it indicates that the data retrieval vector does not match the modal bitmap; when the dot product result is close to 1, it indicates that the data retrieval vector has a high degree of matching with the modal bitmap. Therefore, the size of the dot product can accurately determine the degree of matching between the data retrieval vector and the modal bitmap. The dot product operation is applicable to various types of modal bitmap indexes, including but not limited to binary modal identifiers. According to the specific application scenarios and data characteristics, different modal bitmap indexes can be flexibly selected to represent the modal information of the data to meet different data processing requirements. Furthermore, since the dot product operation is a general and standardized operation, it is easy to integrate with other algorithms and data processing technologies. By combining with other algorithms, the system can further expand its functionality, such as using machine learning algorithms to optimize matching results or using deep learning algorithms to perform more complex processing and analysis of modal data.
[0423] In this embodiment, matching the data retrieval vector with a semantic annotation index table of unstructured multimodal data in a database to retrieve multimodal data matching the data retrieval request includes:
[0424] Based on the data retrieval vector, matching is performed with a semantic annotation index table of unstructured multimodal data in a database to recall the multimodal data;
[0425] From the recalled multimodal data, based on the similarity, some multimodal data are selected as multimodal data matching the data retrieval request for push.
[0426] Figure 2 FIG. 1 is a schematic diagram of a structure of a large model-based unstructured data full-text search device according to this embodiment. Figure 2 As shown, it includes:
[0427] A request acquisition unit 201, used to acquire a data retrieval request;
[0428] A request parsing unit 202, configured to parse the data retrieval request to determine data retrieval content;
[0429] A vectorization unit 203, used to convert the data retrieval content into a data retrieval vector;
[0430] The matching unit 204 is used to match the data retrieval vector with a semantic annotation index table of unstructured multimodal data in a database to retrieve multimodal data matching the data retrieval request.
[0431] For an exemplary explanation of each of the above units, please refer to the above Figure 1 Description.
[0432] Figure 3 Schematic diagram of the structure of the electronic device of this embodiment; the electronic device may include:
[0433] One or more processors 301;
[0434] The computer readable medium 302 may be configured to store one or more programs,
[0435] When the one or more programs are executed by the one or more processors, the one or more processors implement the methods described in the above embodiments.
[0436] Figure 4 is the hardware structure of the electronic device of this embodiment; Figure 4 As shown, the hardware structure of the electronic device may include: a processor 401, a communication interface 402, a computer readable medium 403 and a communication bus 404;
[0437] The processor 401, the communication interface 402, and the computer readable medium 403 communicate with each other via a communication bus 404;
[0438] Optionally, the communication interface 402 may be an interface of a communication module, such as an interface of a GSM module;
[0439] The processor 401 may be specifically configured to implement the method provided in the above embodiment.
[0440] The processor 401 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0441] The computer readable medium 403 can be, but is not limited to, a random access memory medium (RAM), a read-only memory medium (ROM), a programmable read-only memory medium (PROM), an erasable programmable read-only memory medium (EPROM), an electrically erasable read-only memory medium (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
[0442] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code configured to execute the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program configured to be used by or in combination with an instruction execution system, an apparatus or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0443] Computer program code configured to perform the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0444] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions configured to implement the specified logical function. There are specific sequential relationships in the above-mentioned specific embodiments, but these sequential relationships are only exemplary. When the specific implementation is implemented, these steps may be less, more, or the execution order may be adjusted. That is, in some implementations as replacements, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0445] As another aspect, the present application further provides a computer-readable medium having a computer program stored thereon, which implements the method described in the above embodiment when the program is executed by a processor.
[0446] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the device: calls the set membership model to map the input and obtain a fuzzy input, the input includes at least one of the natural environment at sea, the route environment, and the characteristics of the offshore platform, the natural environment at sea includes at least one of visibility, the number of windy days, and the flow velocity of the water flow; the route environment includes at least one of the traffic flow, the route width, and the route grade; the platform characteristics include at least one of the distance between the slipway and the platform layout: calls the set fuzzy reasoning model, performs fuzzy reasoning based on the fuzzy input and obtains a risk factor; defuzzifies according to the risk factor to obtain a risk level; and evaluates the safe distance between the route and the platform according to the risk level.
[0447] The expressions "first", "second", "the first" or "the second" used in various embodiments of the present disclosure may modify various components regardless of order and / or importance, but these expressions do not limit the corresponding components. The above expressions are only configured for the purpose of distinguishing an element from other elements. For example, a first user device and a second user device represent different user devices, although both are user devices. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present disclosure.
[0448] When one element (e.g., a first element) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" or "connected to" another element (e.g., a second element), it is understood that the one element is directly connected to the other element or the one element is indirectly connected to the other element via yet another element (e.g., a third element). Conversely, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (the second element), no element (e.g., a third element) is interposed between the two.
[0449] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A full-text retrieval method for unstructured data based on a large model, characterized in that: Based on the large model, the following steps are performed, including: Get data retrieval request; Parsing the data retrieval request to determine data retrieval content; Converting the data retrieval content into a data retrieval vector; Based on the data retrieval vector, matching is performed with a semantic annotation index table of unstructured multimodal data in a database to retrieve multimodal data matching the data retrieval request; The step of parsing the data retrieval request to determine the data retrieval content includes: Performing semantic extraction on the data retrieval request to obtain multiple context features therein; Based on each context feature, a request description tag is generated; Generate a semantic ontology of the data retrieval request based on request description tags corresponding to all context features; Using the semantic ontology as the data retrieval content; The semantic extraction of the data retrieval request to obtain multiple context features therein includes: Performing semantic extraction on the data retrieval request to obtain semantic entities and neighbor link relationships between semantic entities; Based on the hash mapping between the semantic entities and the neighbor link relationships between the semantic entities, determining that the data retrieval request is to be semantically extracted to obtain a plurality of context features therein; The step of generating a request description tag based on each context feature includes: Encode each context feature to obtain an encoding vector; Analyze the attention weights between each context feature and other context features to obtain an attention weight vector; The encoding vector and the attention weight vector are merged to obtain a request description label corresponding to each context feature; The step of generating the semantic ontology of the data retrieval request based on the request description tags corresponding to all context features includes: Build a knowledge graph based on the request description tags corresponding to all context features; Performing semantic reasoning based on the knowledge graph to generate a semantic ontology of the data retrieval request; Wherein, the request description tags corresponding to all context features are used to construct a knowledge graph. include; Mapping each context feature to an entity in the knowledge graph using each context feature as a key in a key-value pair and a request description tag corresponding to each context feature as a value in the key-value pair; The semantic relationship strength between context features is mapped to the link strength between entities in the knowledge graph.
2. The method according to claim 1, characterized in that The converting the data retrieval content into a data retrieval vector comprises: Dividing the data retrieval content into blocks to obtain a plurality of content blocks; Perform hidden encoding on each content block to obtain the corresponding hidden representation vector; The hidden representation vectors corresponding to all content blocks are aligned, and the aligned hidden representation vectors are weighted merged to obtain the data retrieval vector.
3. The method according to claim 1, characterized in that The matching, based on the data retrieval vector, with a semantic annotation index table of unstructured multimodal data in a database to retrieve multimodal data matching the data retrieval request includes: The semantic similarity between the data retrieval vector and the semantic annotation index table of unstructured multimodal data in the database is calculated to retrieve multimodal data matching the data retrieval request.
4. The method according to claim 3, characterized in that The semantic annotation index table of the multimodal data includes a modality dictionary, and the modality dictionary includes a modality bitmap index and a corresponding semantic annotation index vector: Calculating the semantic similarity between the data retrieval vector and the semantic annotation index table of unstructured multimodal data in the database includes: The data retrieval vector is matched with the modal bitmap index, and a bitwise AND operation is performed with the semantic annotation index vector corresponding to the matched modal bitmap index to calculate the semantic similarity between the data retrieval vector and the semantic annotation index table of unstructured multimodal data in the database.
5. The method according to claim 4, characterized in that The data retrieval vector is matched with the modality bitmap index, including: The data retrieval vector is dot-producted with the binary modality identifier in the modality bitmap index to determine whether the data retrieval vector matches the modality bitmap based on the size of the dot product.
Citation Information
Cited By
A large model application method and system for data retrieval
CN120256544B
