Intelligent Retrieval Method for Multi-Source Data Based on AI and Natural Language Processing

Abstract

This invention discloses an intelligent multi-source data retrieval method based on AI and natural language processing, belonging to the fields of data retrieval and artificial intelligence technology. The method includes: acquiring raw data from heterogeneous data sources, performing semantic parsing and entity extraction, and constructing a multi-source semantic knowledge graph; receiving natural language retrieval requests, performing intent recognition and semantic expansion, and generating at least two semantically expanded retrieval expressions; mapping the semantically expanded retrieval expressions in parallel to subgraphs of the multi-source semantic knowledge graph, performing semantic matching and path exploration, and obtaining an intermediate retrieval result set; based on the semantic correlation between the semantically expanded retrieval expressions, performing cross-validation and weighted fusion on the intermediate retrieval result set to generate a fused retrieval result set. This method, through multi-retrieval expression expansion and knowledge graph semantic matching, solves the problems of semantic heterogeneity in heterogeneous data and retrieval bias, improving the comprehensiveness and accuracy of retrieval in complex query scenarios.

Description

Technical Field

[0001] This invention relates to the fields of data retrieval and artificial intelligence technology, specifically to a multi-source data intelligent retrieval method based on AI and natural language processing. Background Technology

[0002] Existing cross-source data retrieval technologies largely rely on keyword matching or shallow semantic analysis. When faced with structured and unstructured data from different systems, they typically employ independent queries followed by manual aggregation. This approach struggles to understand the implicit semantic relationships within the data. When users express complex search intents in natural language, the system can only output discrete results based on literal matching, failing to recognize different expressions of the same object across multiple sources. This results in highly redundant and poorly correlated search results. Existing solutions often employ metadata mapping or simple ontology alignment strategies when constructing a unified search view, limiting the depth of semantic mining of the original data. Entities expressing the same meaning in different data sources cannot be automatically associated due to differences in description methods, and the strength of relationships between entities is not quantified, causing the search to remain at the string level. In the natural language query processing, existing technologies generally extract keywords directly to generate a single query, ignoring implicit semantic variations in user intent. This results in narrow search result coverage, failing to balance recall and precision.

[0003] Faced with the aggregation of results from multiple sources, existing methods generally use string similarity or classifier scoring for sorting, lacking a verification mechanism for the deep semantic consistency between results from different query dimensions. This easily leads to superficially related but semantically contradictory information being promoted to the top. The connectivity and adaptation latency issues of dynamically accessing multi-source data also increase the maintenance cost of the retrieval system. The problems this invention aims to solve include: how to perform sentence-level semantic decomposition and quantification of relationships in multi-source raw data, and fuse them into a semantic knowledge graph capable of supporting path exploration through configurable mapping rules; how to generate semantically multi-dimensional extended retrieval expressions during the natural language understanding stage, and after parallel graph matching, utilize the semantic correlations between extended expressions to cross-validate and weightedly fuse the results, thereby improving the overall retrieval performance without relying on the accuracy of a single query. Summary of the Invention

[0004] This invention provides an intelligent retrieval method for multi-source data based on AI and natural language processing, aiming to solve the problem that existing multi-source retrieval technologies suffer from insufficient semantic understanding and limited query dimensions, resulting in a difficulty in simultaneously achieving high recall and precision. This method, based on the construction of a multi-source semantic knowledge graph, expands natural language queries into multiple semantically related retrieval expressions. These expressions are explored in parallel within the graph, and the results are cross-validated and fused using the relationships between them, achieving high-precision intelligent retrieval for heterogeneous data.

[0005] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a multi-source data intelligent retrieval method based on AI and natural language processing, which solves the problems of inconsistent semantic understanding of multi-source heterogeneous data, incomplete coverage of retrieval intent, and low confidence of single matching results, and realizes accurate and comprehensive retrieval across data sources.

[0006] This method acquires raw data from at least two heterogeneous data sources, performs semantic parsing and entity extraction on the raw data, and constructs a multi-source semantic knowledge graph. By performing unified semantic modeling on the multi-source data, raw data with different structures are transformed into graph structures with entities, relationships, and attributes, providing a consistent semantic foundation for subsequent cross-source retrieval. After receiving an initial retrieval request input by a user in natural language, the method performs intent recognition and semantic expansion on the initial retrieval request, generating at least two semantically expanded retrieval expressions. Through natural language intent recognition and semantic expansion of core words, the lexical gap between user expression and data representation can be bridged, covering more potential retrieval dimensions without relying on the user's precise query construction. The generated at least two semantically expanded retrieval expressions are mapped in parallel to the corresponding subgraphs of the multi-source semantic knowledge graph, and semantic matching and path exploration are performed to obtain the intermediate retrieval result set corresponding to each semantically expanded retrieval expression. Utilizing the relational path exploration capabilities of the knowledge graph, the retrieval process is not limited to direct entity matching, but can also discover deep associations along semantic relationships, improving recall and semantic completeness. Based on the semantic relevance between at least two semantically extended search terms, the intermediate search result sets are cross-validated and weighted to generate a fused search result set. This approach strengthens high-confidence results by mutually verifying multiple extended terms, suppressing noise or ambiguous matching that might be introduced by a single search term, and improving the accuracy and reliability of the search results.

[0007] Preferably, when constructing a multi-source semantic knowledge graph, for each piece of original data from each heterogeneous data source, dependency parsing in natural language processing is used to identify the subject-verb-object structure and modification relations; the subject and object are treated as entity nodes, and the predicate is treated as the initial relation edges between entity nodes; the attributes of entity nodes are filled according to the modification relations, and the initial relation edges are assigned initial weights according to the confidence of dependency parsing, resulting in a source-internal semantic subgraph for each heterogeneous data source; through preset ontology mapping rules, entity nodes representing the same real-world object in different source-internal semantic subgraphs are semantically aligned and merged, thereby fusing all source-internal semantic subgraphs into a multi-source semantic knowledge graph. By constructing source-internal subgraphs based on syntactic analysis, unstructured text is automatically transformed into a structured semantic network, and ontology mapping is used to achieve cross-source entity unification, ensuring the construction quality and semantic consistency of the multi-source knowledge graph.

[0008] As a technical solution of this invention, the preset ontology mapping rules include direct mapping rules based on string similarity, similarity mapping rules based on semantic vectors, and equivalence class mapping rules based on a predefined ontology library. The combined use of multiple mapping rules can avoid alignment omissions or errors caused by a single matching standard, improving the robustness and accuracy of entity alignment.

[0009] Preferably, the process of intent recognition and semantic expansion for the initial search request specifically involves: segmenting and tagging the initial search request with words to extract the core verbs and core noun phrases; inputting the core noun phrases into a pre-trained semantic vector model to retrieve at least two synonyms or near-synonyms in the semantic vector space whose vector distance to the core noun phrases is less than a preset threshold; combining the core verbs with each synonym or near-synonym, and using the combined phrases as semantically expanded search terms. This allows for the controlled expansion of multiple semantically similar but differently expressed search terms while preserving the user's core intent, thus enriching the search entry points.

[0010] Furthermore, the specific process of mapping at least two semantically expanded search terms to corresponding subgraphs of the multi-source semantic knowledge graph in parallel is as follows: The phrases in each semantically expanded search term are decomposed into search entities and search relations; for each semantically expanded search term, a set of candidate entity nodes matching the search entity is located in the multi-source semantic knowledge graph; starting from each node in this set of candidate entity nodes, a breadth-first traversal is performed in the multi-source semantic knowledge graph along the direction indicated by the search relations, collecting the nodes and edges encountered during the traversal to form an intermediate search result set corresponding to each semantically expanded search term. This approach, through the structured decomposition of search terms and graph traversal, accurately associates natural language search requirements with local semantic regions of the knowledge graph, achieving focused and interpretable retrieval.

[0011] When performing a breadth-first traversal, the maximum traversal depth is dynamically limited based on the path length between the current node and the target node, as well as the sum of the weights of all relational edges along the path. This dynamic depth limitation avoids search bloat caused by an excessively large graph, while preserving deeply relevant paths, thus ensuring a balance between retrieval efficiency and result quality.

[0012] When performing entity matching, the edit distance between the string of the retrieved entity and the name string of each entity node in the multi-source semantic knowledge graph is calculated, and the cosine similarity between the semantic vector of the retrieved entity and the semantic vector of each entity node is calculated. Entity nodes that simultaneously satisfy an edit distance less than a first preset value and a cosine similarity greater than a second preset value are included in the candidate entity node set. When the candidate entity node set is empty, a fuzzy expansion mechanism is triggered, splitting the retrieved entity into several sub-word units, matching them separately, and then merging the matching results to form the candidate entity node set. This dual-path matching and fuzzy expansion mechanism takes into account both literal similarity and deep semantic similarity, effectively addressing naming differences and expression variations in the data, and avoiding retrieval interruptions caused by no results from strict matching.

[0013] Preferably, the process of cross-validation and weighted fusion of the intermediate search result set specifically involves: calculating the semantic similarity between the first result item in the first intermediate search result set corresponding to the first semantic expansion search expression and the second result item in the second intermediate search result set corresponding to the second semantic expansion search expression; when the semantic similarity exceeds a preset validation threshold, the first result item and the second result item are marked as a pair of mutually validated result items; a first fusion weight higher than that of the result items that failed validation is assigned to the pair of mutually validated result items; and the result items in all intermediate search result sets are sorted and deduplicated according to the first fusion weight to generate a fused search result set. Utilizing cross-validation of the same information using multiple search expressions allows evidence from different expanded search paths to support each other, significantly improving the credibility of the results, and resulting in a fused ranking that better reflects the user's actual needs.

[0014] As a specific implementation of this invention, the process of calculating the semantic similarity between two result items is as follows: extract the first subgraph corresponding to the first result item in the multi-source semantic knowledge graph, and the second subgraph corresponding to the second result item in the multi-source semantic knowledge graph; calculate the graph edit distance between the first and second subgraphs, which is the minimum number of operations required to transform the first subgraph into the second subgraph through adding, deleting, and modifying nodes and edges; and use the reciprocal of the graph edit distance as the semantic similarity between the first and second result items. Measuring semantic similarity based on subgraph structural differences can more fully reflect the contextual relationships and complete semantics of the result items in the knowledge graph, and has stronger discriminative power compared to comparing only entity-level indicators.

[0015] As a preferred implementation, after generating the fused search result set, the process further includes a step of dynamically reordering the fused search result set based on feedback: receiving user click or browsing behavior data for at least one result item in the fused search result set; extracting the target result item that the user is interested in based on the click or browsing behavior data; calculating the content relevance between the target result item and other unclicked result items in the fused search result set; and increasing the ranking position of unclicked result items whose relevance to the target result item is higher than a preset reordering threshold based on the content relevance. By dynamically capturing real-time user feedback and adjusting the ranking in a timely manner, results that users are potentially interested in can be quickly prioritized without interrupting the search experience, thereby improving interaction efficiency and user satisfaction.

[0016] In another preferred embodiment, after generating the fused search result set, a personalized filtering step based on user profile is further included: obtaining the identity identifier of the user who initiated the initial search request, extracting the user's long-term interest vector and short-term behavior vector from the user profile database; mapping each result item in the fused search result set to a result semantic vector; calculating the first matching degree between each result semantic vector and the long-term interest vector, and the second matching degree between each result semantic vector and the short-term behavior vector; filtering the result items in the fused search result set based on the weighted sum of the first and second matching degrees, removing result items whose weighted sum is below the personalization threshold. By comprehensively considering long-term preferences and recent immediate needs, content irrelevant to the user's personality is filtered out, making the final results more consistent with individual cognition and usage scenarios.

[0017] As a preliminary technical solution of this invention, before acquiring raw data from at least two heterogeneous data sources, a multi-source data dynamic access channel is constructed: an independent connector is configured for each heterogeneous data source, and this connector contains the data reading protocol and authentication information of the heterogeneous data source; the central scheduler polls the status of each connector according to a preset time period, and when any connector is ready, the central scheduler triggers the connector to perform a data retrieval operation; the connector encapsulates the retrieved raw data into a unified format data message and pushes it to a shared data buffer for subsequent steps to retrieve raw data from the shared data buffer. Through a unified access framework, the differences between multi-source data are shielded at the connector layer, ensuring the real-time nature and consistency of data acquisition, and supporting the upper-layer dynamic knowledge graph construction and continuous retrieval capabilities.

[0018] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0019] By identifying subject-verb-object structures in the original data through dependency parsing, the subject and object are treated as entity nodes, and the predicate as initial relation edges. Edge weights are assigned based on modification relations and dependency parsing confidence, constructing an intra-source semantic subgraph for each heterogeneous data source. Utilizing ontology mapping rules that include string similarity, semantic vector similarity, and predefined ontology equivalence classes, entity nodes representing the same real-world object in different subgraphs are semantically aligned and merged, thus forming a multi-source semantic knowledge graph. The entity relations and attributes within this graph are all weighted, capable of carrying multi-granular information from syntactic to semantic levels. This allows subsequent retrieval to conduct deep semantic exploration based on paths and weights within the graph, rather than remaining at the surface-level key-value matching. Directly extracting semantic structures from sentence parsing and aligning entities across sources eliminates representational differences between heterogeneous data, reducing result isolation and mismatches caused by inconsistent data formats and naming. Upon receiving a natural language retrieval request, core verbs and core noun phrases are extracted. Synonyms or near-synonyms with the core noun phrases, whose distance is less than a preset threshold, are retrieved using a semantic vector model. The core verbs and these phrases are combined into at least two semantically expanded retrieval expressions. Each expanded retrieval expression is then broken down into retrieval entities and retrieval relations. Candidate entity node sets are located in a multi-source semantic knowledge graph, and a breadth-first traversal is performed along the retrieval relation direction to obtain the corresponding intermediate retrieval result set. The graph edit distance between the corresponding intermediate result items of different expanded retrieval expressions and their subgraphs in the graph is calculated. The reciprocal of this distance is used as the semantic similarity. Result items exceeding the verification threshold are marked as mutually verified and assigned a higher first fusion weight than those that failed. Based on this, all intermediate retrieval results are sorted and deduplicated to generate a fused retrieval result set. Multiple expanded retrieval expressions cover the same query intent from different semantic perspectives, capturing variations of implicit user needs and avoiding missed detections caused by expression biases in a single query. After generating multidimensional intermediate results through parallel graph matching, a cross-validation mechanism based on subgraph edit distance is employed. This ensures that content with consistent semantic support from different extended dimensions receives higher ranking weights, unaffected by noise from any single dimension. It enhances the relevance and credibility of results without relying on the absolutely optimal construction of the query, while suppressing false recalls caused by homonyms or ambiguities. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0021] Figure 1 This is a flowchart of a multi-source data intelligent retrieval method;

[0022] Figure 2This is a flowchart of the multi-source semantic knowledge graph construction process;

[0023] Figure 3 This is a flowchart of a retrieval-based method for generating queries based on semantic expansion.

[0024] Figure 4 This is a flowchart of semantic expansion retrieval mapping and path exploration in a multi-source semantic knowledge graph;

[0025] Figure 5 This is a flowchart of cross-validation and weighted fusion of search results based on graph edit distance;

[0026] Figure 6 It is a flowchart of dynamic feedback rearrangement and personalized filtering of integrated search results;

[0027] Figure 7 This is a distribution map of the cosine distance between synonyms and near-synonyms in semantically expanded search queries. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] See Figure 1 This invention provides a multi-source data intelligent retrieval method based on AI and natural language processing. The overall implementation scheme of the method is as follows: Obtain the original data corresponding to at least two heterogeneous data sources, and perform semantic parsing and entity extraction on the original data to construct a multi-source semantic knowledge graph. After receiving an initial retrieval request input by a user in natural language form, perform intent recognition and semantic expansion on the initial retrieval request to generate at least two semantically expanded retrieval expressions. Map the at least two semantically expanded retrieval expressions in parallel to the corresponding subgraphs of the multi-source semantic knowledge graph, perform semantic matching and path exploration, and obtain an intermediate retrieval result set corresponding to each semantically expanded retrieval expression. Based on the semantic correlation between the at least two semantically expanded retrieval expressions, perform cross-validation and weighted fusion on the intermediate retrieval result sets to generate a fused retrieval result set.

[0030] Example 1:

[0031] In specific implementation, please refer to Figure 2 The process of performing semantic parsing and entity extraction on the original data and constructing a multi-source semantic knowledge graph includes two stages: construction of intra-source semantic subgraphs and cross-source semantic alignment and fusion.

[0032] In the source semantic subgraph construction stage, for each piece of raw data from each heterogeneous data source, dependency parsing tools from natural language processing are used to process the raw data. The dependency parsing tool performs word segmentation, part-of-speech tagging, and dependency arc tagging on each sentence in the raw data, outputting a dependency parsing tree. In the dependency parsing tree, subject, predicate, and object components are identified, along with modifying relationships such as attributive, adverbial, and complement. Words corresponding to subject components are designated as subject entities, and words corresponding to object components are designated as object entities. Both subject and object entities are treated as entity nodes. Words corresponding to predicate components are used as initial relation edges between entity nodes.

[0033] In a dependency parsing tree, modification relations include attributives modifying the subject or object, adverbs modifying the predicate, and complements providing supplementary information about the predicate. The attributes of entity nodes are populated according to the modification relations. Specifically: if the subject or object entity is modified by an attributive, the corresponding word or phrase is filled into the attribute key-value pair of the corresponding entity node. The attribute name is determined by the modification type, and the attribute value is the attributive text. If the predicate is modified by an adverb, the adverbial information is recorded as an additional attribute of the initial relation edge. If the predicate is modified by a complement, the complement information is recorded as an additional attribute of the initial relation edge. Along with outputting the dependency parsing tree, dependency parsing also outputs a confidence score for each dependency arc. This confidence score is a real number between 0 and 1, representing the degree of confidence the dependency parsing tool has in its judgment of that dependency relation. Initial weights are assigned to the initial relation edges based on the confidence scores from dependency parsing. Specifically, the confidence score corresponding to the dependency relationship between the predicate and the subject is used as the first confidence score, and the confidence score corresponding to the dependency relationship between the predicate and the object is used as the second confidence score. The average of the first and second confidence scores is calculated, and this average is used as the initial weight for the relation edge with the predicate as the relation edge. Alternatively, the smaller value between the first and second confidence scores can be directly used as the initial weight for the initial relation edge. After the above processing, each piece of original data yields several entity nodes, the attributes of the entity nodes, the initial relation edges between the entity nodes, and the initial weights of the initial relation edges. After all the original data within the same heterogeneous data source has been processed, these entity nodes, entity node attributes, initial relation edges between entity nodes, and initial weights of the initial relation edges are merged to form the source semantic subgraph of the heterogeneous data source.

[0034] After constructing the intra-source semantic subgraphs for all heterogeneous data sources, the cross-source semantic alignment and fusion stage begins. Using predefined ontology mapping rules, entity nodes representing the same real-world object in different intra-source semantic subgraphs are semantically aligned and merged, thereby fusing all intra-source semantic subgraphs into a multi-source semantic knowledge graph. The predefined ontology mapping rules include direct mapping rules based on string similarity, similarity mapping rules based on semantic vectors, and equivalence class mapping rules based on a predefined ontology library.

[0035] The direct mapping rule based on string similarity is as follows: Obtain the name string of the first entity node in the first source semantic subgraph and the name string of the second entity node in the second source semantic subgraph. Calculate the edit distance between the name strings of the first and second entity nodes. Convert the edit distance into string similarity through normalization. The normalization method is to take the larger of the lengths of the name strings of the first and second entity nodes as the baseline length. The string similarity is equal to 1 minus the quotient of the edit distance divided by the baseline length. When the string similarity exceeds a preset string mapping threshold, the first and second entity nodes are determined to represent the same real-world object. The preset string mapping threshold can be set to 0.8.

[0036] The semantic vector-based similarity mapping rule works as follows: The name string of the first entity node is input into a pre-trained semantic vector model, which outputs the first semantic vector corresponding to the first entity node. The name string of the second entity node is input into the same semantic vector model, which outputs the second semantic vector corresponding to the second entity node. The cosine similarity between the first and second semantic vectors is calculated. When the cosine similarity exceeds a preset vector mapping threshold, the first and second entity nodes are determined to represent the same real-world object. The preset vector mapping threshold can be set to 0.85. The semantic vector model is a distributed semantic representation model trained on a large-scale corpus, such as the Skip-gram model of Word2Vec or the BERT model. It can map words or phrases to a dense vector space of fixed dimensions, where semantically similar words are close in distance within the vector space.

[0037] The equivalence class mapping rules based on a predefined ontology library are as follows: An ontology library is pre-built, defining multiple ontology classes, each containing several equivalent entity names. When the name strings of a first entity node and a second entity node both match equivalent entity names within the same ontology class in the ontology library, it is determined that the first entity node and the second entity node represent the same real-world object. The equivalent entity names in the ontology library are sourced from manual annotation and imports from external knowledge bases.

[0038] During the mapping process, it can be understood that the three mapping rules can be combined. The combination method is as follows: when both the direct mapping rule based on string similarity and the similarity mapping rule based on semantic vectors determine that two entity nodes match, and the equivalence class mapping rule based on the predefined ontology does not indicate a conflict, the two entity nodes are semantically aligned and merged. Alternatively, a priority can be assigned to each of the three mapping rules, with the priority order from highest to lowest as follows: equivalence class mapping rule based on the predefined ontology, similarity mapping rule based on semantic vectors, and direct mapping rule based on string similarity. When a high-priority rule produces a matching result, the result of the high-priority rule is directly adopted; if the high-priority rule has no matching result, the lower-priority rules are used sequentially.

[0039] The entity node merging process is as follows: multiple entity nodes that are semantically aligned to be the same real-world object are merged into a single fused entity node. The name of the fused entity node is the middle string length of the original entity node names, or a standard name defined in the predefined ontology library. The attribute set of the fused entity node is the union of the attributes of the merged entity nodes. If the same attribute key has values ​​in multiple entity nodes, the attribute value from the source semantic subgraph with higher confidence is retained. The confidence level is determined by the reliability level of the original data source. The initial relation edges associated with the fused entity node are redirected to the fused entity node according to the structure of the source semantic subgraph, while the initial weights of the initial relation edges remain unchanged. After all entity nodes in the source semantic subgraphs are merged, a unified multi-source semantic knowledge graph is formed. The multi-source semantic knowledge graph contains the merged entity nodes, entity node attributes, relation edges, and relation edge weights. Subsequent retrieval operations are performed on the multi-source semantic knowledge graph.

[0040] Example 2:

[0041] In specific implementation, please refer to Figure 3 The process of identifying intent and semantically expanding the initial search request and generating at least two semantically expanded search terms includes three steps: core component extraction, synonym and near-synonym phrase retrieval, and combination of semantically expanded search terms.

[0042] The initial search request is text input by the user in natural language, with no limit on text length. The initial search request is segmented and tagged with parts of speech (PTS). A joint segmentation and PTS model based on conditional random fields is used to process the initial search request string, obtaining a word sequence and the PTS tag for each word. Based on the PTS tagging results, core verbs and core noun phrases are extracted. Core verbs are extracted by traversing the word sequence and selecting words tagged with verbs that are located at the predicate center of the sentence. The identification of the predicate center position is achieved through dependency parsing. The initial search request is subjected to dependency parsing to obtain a dependency syntax tree. In the dependency syntax tree, verbs that are direct children of the root node and have a core dependency relationship are defined as core verbs. If there is no core relation verb among the direct children of the root node, the verb with the shortest path to the root node is selected as the core verb. The method for extracting core noun phrases is as follows: In the dependency syntax tree, the subject and object components of the core verb are identified. The noun phrases corresponding to the subject and the object components are defined as core noun phrases. If the core verb has no subject and no object components, then noun phrases with part-of-speech tags of nouns and dependency relations of non-modifier components are selected as core noun phrases. The internal structure of the noun phrases includes single nouns, multiple nouns in parallel, or combinations of adjectives and nouns.

[0043] After obtaining the core noun phrases, they are input into a pre-trained semantic vector model. The semantic vector model maps the core noun phrases to a fixed-dimensional real-valued vector, called the core noun phrase semantic vector. The semantic vector model is a pre-trained language representation model based on the Transformer architecture. The model structure includes a multi-layer bidirectional Transformer encoder. Each Transformer encoder layer contains a multi-head self-attention sublayer and a feedforward neural network sublayer. The multi-head self-attention sublayer uses 12 attention heads, and the hidden layer dimension of the feedforward neural network sublayer is 3072. The input layer of the semantic vector model receives the word sequence after word segmentation, and the output layer outputs a 768-dimensional vector at a specific [CLS] marker position as the semantic representation of the entire input text. The training process of the semantic vector model is as follows: a large-scale unlabeled natural language text corpus is collected, and a pre-training task is constructed, which includes a masking language model task and a next-sentence prediction task. The masking language model task randomly masks 15% of the words in the input text and uses context information to predict the masked words. The next-sentence prediction task determines whether two input sentences are consecutive context sentences. The model updates its parameters by maximizing the likelihood function of the pre-training task. The optimizer used is Adam, with an initial learning rate of 1e-4, a batch size of 256, and 1 million training iterations. After training, the model is capable of mapping any natural language text into semantic vectors. For core noun phrases, these are used as input text to the semantic vector model, and the 768-dimensional vector corresponding to the [CLS] marker position in the model's output layer is taken as the semantic vector of the core noun phrase.

[0044] Retrieve at least two synonyms or near-synonyms from the core noun phrase in the semantic vector space, provided their vector distance to the core noun phrase is less than a preset threshold. The semantic vector space consists of semantic vectors obtained by mapping all words and phrases in a predetermined vocabulary set using a semantic vector model. The predetermined vocabulary set is sourced from general dictionaries, domain terminology databases, and high-frequency phrases from historical search logs. Vector distance is defined as cosine distance, calculated using the following formula:

[0045] in, This represents the semantic vector of the core noun phrases, with a dimension of 768. This represents a semantic vector of candidate word groups in a predefined vocabulary set, with a dimension of 768. For the dot product operation of two vectors, Representing vectors L2 norm, Representing vectors L2 norm, The value is the cosine distance, ranging from [0,2]. The preset threshold is set to 0.3. The basis for setting the preset threshold to 0.3 is as follows: on a dataset of 500 manually labeled pairs of synonyms and near-synonyms, the cosine distance distribution between the pairs of synonyms and near-synonyms is calculated, and the upper quartile of the distribution is taken as the boundary for distinguishing between synonyms, near-synonyms and non-synonyms. The upper quartile is 0.32, and the value is rounded down to 0.3.

[0046] The retrieval process is as follows: Each candidate word group in the predetermined vocabulary set is traversed. The candidate word group is input into a semantic vector model to obtain its semantic vector. The cosine distance between the semantic vector of the core noun phrase and the semantic vector of the candidate word group is calculated. If the cosine distance is less than 0.3, the candidate word group is assigned to a set of synonyms or near-synonyms. From the set of synonyms or near-synonyms, at least two words with the smallest cosine distance are selected. If the size of the set of synonyms or near-synonyms is less than 2, all words are directly used, but at least one extended word is guaranteed to be generated. When the set of synonyms or near-synonyms is empty, the core noun phrase itself is used as the unique extended word. Finally, at least two synonyms or near-synonyms are obtained, including the core noun phrase itself as the baseline extended word, and several extended word groups with the smallest cosine distance.

[0047] Optionally, in order to control the direction of expansion, part-of-speech filtering can be introduced to retain only candidate word groups that have the same part of speech as the core noun phrase.

[0048] After obtaining at least two synonyms or near-synonyms, the core verb is combined with each synonym or near-synonym. The combination method is as follows: the core verb is placed before or after the synonym or near-synonym, the position determined by the relative order of the core verb and the core noun phrase in the initial search request. Specifically: if the core verb precedes the core noun phrase, the combined phrase order is core verb first, then synonym or near-synonym; if the core verb follows the core noun phrase, the combined phrase order is synonym or near-synonym first, then core verb. If other words exist between the core verb and the core noun phrase, these other words are inserted as fixed modifiers in the corresponding positions of the combined phrases. The combined phrases are the semantically expanded search queries. Each synonym or near-synonym combined with the core verb generates one semantically expanded search query, thus generating at least two semantically expanded search queries.

[0049] See Figure 7 The figure shows a histogram of the frequency distribution of cosine distance between candidate word groups and core noun phrases in the semantic vector space. The horizontal axis represents the cosine distance, with a value ranging from 0 to 2, and the vertical axis represents the frequency of word groups within the corresponding cosine distance interval. Two different fill styles are used in the figure to represent "synonymous or near-synonymous word groups (cosine distance < 0.3)" and "non-synonymous word groups".

[0050] The figure shows a vertical dashed line marking a cosine distance threshold of 0.3, which is used to distinguish between synonyms or near-synonyms and non-synonyms. The frequency distribution of synonyms or near-synonyms to the left of the threshold shows a highly concentrated trend, with the maximum frequency approaching 70. Furthermore, the cosine distances of most synonyms or near-synonyms are clustered between 0 and 0.15, reflecting that the distances between the semantic vectors of synonyms and near-synonyms are generally small, indicating high semantic similarity.

[0051] In contrast, the frequency distribution of non-synonymous phrases to the right of the threshold is more dispersed, with the cosine distance mainly concentrated in the range of 0.8 to 2.0, and the highest frequency peak located between 1.5 and 1.7. This indicates that the non-synonymous phrases are relatively far from the core noun phrases in the semantic space, and their semantic relevance is weak. The overall frequency of non-synonymous phrases shows a right-skewed distribution, with the frequency gradually decreasing as the cosine distance increases.

[0052] This figure illustrates the rationality and effectiveness of using a cosine distance threshold of 0.3 to filter synonyms and near-synonyms in Embodiment 2 of the present invention. The threshold of 0.3 clearly distinguishes the cosine distance between core noun phrases and synonyms or near-synonyms from non-synonyms, ensuring that the expanded phrases selected in the semantic expansion search have high semantic relevance, which is beneficial for improving the accuracy and recall rate of intelligent retrieval of multi-source data.

[0053] Example 3:

[0054] In specific implementation, please refer to Figure 4 The process involves mapping at least two semantically expanded retrieval expressions in parallel to the corresponding subgraphs of a multi-source semantic knowledge graph, performing semantic matching and path exploration, and obtaining the intermediate retrieval result set corresponding to each semantically expanded retrieval expression. It includes four parts: phrase splitting, candidate entity node localization, fuzzy expansion processing, and breadth-first traversal.

[0055] Each semantically expanded search phrase in at least two semantically expanded search expressions is split into search entities and search relations. The splitting method is as follows: dependency parsing is performed on the semantically expanded search phrase to obtain a dependency syntax tree. Core verbs and core noun phrases are identified in the dependency syntax tree. The core noun phrases are used as search entities, and the core verbs are used as search relations. If there is no verb in the semantically expanded search phrase, the entire phrase is used as the search entity, and the search relation is marked as a general association relation. Optionally, a rule-based splitting method can also be used, matching verbs in the semantically expanded search phrases using a pre-built verb dictionary, using the noun phrases preceding or following the verb as search entities, and the verb itself as the search relation.

[0056] For each semantically expanded retrieval query, locate the set of candidate entity nodes that match the retrieved entity in the multi-source semantic knowledge graph. The specific steps for locating the set of candidate entity nodes are as follows:

[0057] The edit distance between the string of the retrieved entity and the name string of each entity node in the multi-source semantic knowledge graph is calculated. Edit distance is defined as the minimum number of single-character edit operations required to convert one string into another; edit operations include insertion, deletion, and replacement. In this step, the edit distance is calculated using the Levenstein distance algorithm. After obtaining the edit distance, it is normalized to obtain the string similarity. The normalization method is as follows: the larger value between the length of the retrieved entity's string and the length of the entity node's name string is taken as the baseline length, and the edit distance is divided by the baseline length to obtain the normalized edit distance. The judgment condition is set as an edit distance less than a first preset value. The first preset value is set to 3, based on the fact that statistical analysis of the character difference distribution of synonymous entity names on an open-source named entity recognition dataset shows that the edit distance between more than 95% of synonymous entity names does not exceed 3. Therefore, an edit distance less than 3 is used as the matching condition at the string level.

[0058] The cosine similarity between the semantic vector of the retrieved entity and the semantic vector of each entity node in the multi-source semantic knowledge graph is calculated. The semantic vector of the retrieved entity is obtained by inputting the string of the retrieved entity into the semantic vector model. The semantic vectors of the entity nodes are pre-calculated and stored in the attributes of the entity nodes when constructing the multi-source semantic knowledge graph. The semantic vector model is the same model used to generate the semantically expanded retrieval expression. The cosine similarity calculation formula is the dot product between the semantic vector of the retrieved entity and the semantic vector of the entity node divided by the product of the L2 norms of the two vectors. The judgment condition is set as the cosine similarity being greater than a second preset value. The second preset value is set to 0.8. The basis for setting the second preset value to 0.8 is: on the semantic similarity manually labeled dataset, the lower limit of the similarity distribution of semantic equivalence class samples is taken. It has been verified that the lower limit is 0.78, and it is rounded up to 0.8 to ensure high semantic matching accuracy.

[0059] Entity nodes that simultaneously satisfy both an edit distance less than a first preset value and a cosine similarity greater than a second preset value are included in the candidate entity node set. If an entity node satisfies only one of these conditions, it is not included in the candidate entity node set. Optionally, the edit distance and cosine similarity between all entity nodes in the multi-source semantic knowledge graph and the retrieved entity can be calculated in parallel, and their intersection can be taken.

[0060] When the candidate entity node set is empty, a fuzzy expansion mechanism is triggered. The specific steps of this mechanism are as follows: the retrieved entity is split into several sub-word units. The sub-word unit splitting method uses a maximum matching segmentation algorithm based on a vocabulary, dividing the string of the retrieved entity into several longest substrings that match words in a predefined vocabulary. The predefined vocabulary consists of Chinese words and characters in a multi-source semantic knowledge graph whose frequency exceeds a preset frequency threshold, which is set to 5. After segmentation, several sub-word units are obtained. For each sub-word unit, a candidate entity node localization operation is performed, which calculates the edit distance between the sub-word unit's string and the name string of each entity node in the multi-source semantic knowledge graph, as well as the cosine similarity between the semantic vector of the sub-word unit and the semantic vector of the entity node. Entity nodes that simultaneously satisfy an edit distance less than a first preset value and a cosine similarity greater than a second preset value are considered the matching results for the corresponding sub-word unit. Next, the matching results of all sub-word units are merged. The merging method is to take the union of the entity nodes matched by all sub-word units. If an entity node is matched by multiple sub-word units, then that entity node is retained in the merged result, and its matching score is increased. The matching score is equal to the number of sub-word units that match that entity node. The merged set of entity nodes is then used as the candidate entity node set.

[0061] Starting with each node in the candidate entity node set, a breadth-first traversal is performed in the multi-source semantic knowledge graph along the direction indicated by the retrieval relation. The direction indicated by the retrieval relation is the type of relation edge in the multi-source semantic knowledge graph. This relation edge type is semantically aligned with the retrieval relation. The alignment method is to calculate the edit distance between the retrieval relation string and all relation edge type strings in the multi-source semantic knowledge graph, and take the relation edge type with the smallest edit distance as the direction indicated by the retrieval relation. The breadth-first traversal uses a queue data structure. The starting node is added to the queue and marked as visited. Then, nodes are continuously removed from the queue, and neighboring nodes connected to that node through the direction indicated by the retrieval relation are checked. If a neighboring node has not been visited, it is added to the queue and marked as visited. The traversed nodes and connecting relation edges are collected sequentially into the traversal result set. The collected nodes and edges during the traversal process form the intermediate retrieval result set corresponding to the semantically expanded retrieval expression.

[0062] When performing a breadth-first traversal, the maximum traversal depth is dynamically limited based on the path length between the current node and the target node, and the sum of the weights of all relation edges along the path. The target node is defined as a node on the traversal path that satisfies the entity retrieval constraints; that is, a node explored in the graph after starting from the candidate entity node set, and whose semantics are related to the target implicit in the initial retrieval request. The dynamically limited maximum depth is determined by the following formula:

[0063]

[0064] in, This represents the maximum remaining depth that allows traversal to continue from the current position. It is an integer value, and its range is an integer greater than or equal to 1. This represents the sum of the weights of all relation edges along the path from the starting node to the current node. The weights are the initial weights of the relation edges labeled in the multi-source semantic knowledge graph, and their values ​​are between 0 and 1. This represents the number of edges traversed from the starting node to the current node, i.e., the current depth, and is a positive integer. This is the path length penalty coefficient, with a value of 0.2. The basis for choosing 0.2 is: based on the typical decay rate of the weights of relation edges in the graph, the contribution of each additional hop in the path length to the overall cost is set to be equivalent to a weight increment of 0.2. The preset depth threshold is set to 3.0. The rationale for choosing 3.0 is as follows: Traversal experiments on a multi-source semantic knowledge graph revealed that when the overall cost exceeds 3.0, the semantic relevance between subsequent nodes and the starting point significantly decreases. Therefore, 3.0 is set as the upper limit of the traversal constraint. When the calculated... If the value is less than 1, the traversal will not continue along the current path.

[0065] During the traversal, before adding a neighbor node to the queue, calculate the new path after adding the neighbor node. and Then calculate according to the above formula. ,like If the value is greater than or equal to 1, the neighbor node is allowed to be added to the queue and the process continues; otherwise, the neighbor node is not added to the queue, and further exploration of the path is terminated. This implements a dynamic traversal depth limit based on the sum of the path length and weight, avoiding unrestrained expansion across the entire graph.

[0066] Example 4:

[0067] In specific implementation, please refer to Figure 5 Based on the semantic relevance between at least two semantically extended search terms, the process of cross-validating and weightedly fusing intermediate search result sets to generate a fused search result set includes three steps: semantic similarity calculation, mutual validation labeling and weight allocation, and result sorting and deduplication.

[0068] Calculate the semantic similarity between the first result item in the first intermediate search result set corresponding to the first semantic expansion search expression and the second result item in the second intermediate search result set corresponding to the second semantic expansion search expression. The first result item is an element in the first intermediate search result set, and the second result item is an element in the second intermediate search result set. Each result item is contained in a subgraph structure consisting of a set of nodes and relation edges in the multi-source semantic knowledge graph.

[0069] The first subgraph corresponding to the first result item and the second subgraph corresponding to the second result item in the multi-source semantic knowledge graph are extracted. The subgraph extraction method is as follows: starting from the core node corresponding to the result item in the multi-source semantic knowledge graph, a breadth-first traversal is performed along the relation edges within a two-hop range. All nodes and relation edges traversed during the traversal are collected, forming a subgraph centered on the core node. The core node is the entity node corresponding to the endpoint of the traversal path located by candidate entity nodes during the generation of the result item, and is explicitly recorded in the intermediate retrieval result set. The first subgraph structure includes a first set of nodes, a first set of relation edges, and attribute information for each node and weight information for each relation edge. The second subgraph structure includes a second set of nodes, a second set of relation edges, and attribute information for each node and weight information for each relation edge.

[0070] Calculate the graph edit distance between the first and second subgraphs. The graph edit distance is defined as the minimum number of operations required to transform the first subgraph into the second subgraph through adding, deleting, and modifying nodes and edges. The specific calculation method for the graph edit distance is as follows: define the operation cost for node operations and edge operations. A node insertion operation adds a node to the first subgraph, making it correspond to a node in the second subgraph, with a cost of 1. A node deletion operation deletes a node from the first subgraph, with a cost of 1. A node replacement operation modifies the attribute of a node in the first subgraph to the attribute of the corresponding node in the second subgraph, with a cost equal to the dissimilarity of the two node attribute sets. The dissimilarity is calculated as the Jaccard distance between the two node attribute sets, with a value between 0 and 1. The edge insertion operation adds a relation edge to the first subgraph, with a cost of 1. The edge deletion operation deletes a relation edge from the first subgraph, with a cost of 1. The edge replacement operation modifies the weight and type of a relation edge in the first subgraph to match the weight and type of the corresponding relation edge in the second subgraph. The cost is the weighted sum of the edge type difference and the edge weight difference. The edge type difference is 0 when relation edges are of the same type and 1 when they are different. The edge weight difference is the absolute value of the difference between the weights of two relation edges, with weighting coefficients of 0.5 and 0.5 respectively. The graph edit distance is calculated using the A* search algorithm, which searches the state space for the sequence of operations from the first subgraph to the second subgraph. The evaluation function is the sum of the cumulative cost of the executed operations and the estimated remaining cost. The estimated remaining cost uses the number of unmatched nodes and edges as a heuristic estimate. The search continues until the sequence of operations with the minimum cost is found, and the minimum cumulative cost is taken as the graph edit distance.

[0071] After obtaining the graph edit distance, the reciprocal of the graph edit distance is used as the semantic similarity between the first and second result items. The formula for calculating semantic similarity is:

[0072]

[0073] in, Indicates the first result item. Indicates the second result item; This represents the first subgraph corresponding to the first result item in the multi-source semantic knowledge graph. This represents the second subgraph corresponding to the second result item in the multi-source semantic knowledge graph. This indicates that the first subgraph Transform into the second subgraph The required graph edit distance is a non-negative real number. This represents the semantic similarity between the first and second result items. Its value is a real number greater than 0 and less than or equal to 1. The semantic similarity reaches its maximum value of 1 when the graph edit distance is 0, and approaches 0 when the graph edit distance approaches infinity. Adding 1 to the denominator avoids the case where the denominator is zero and also normalizes the similarity value range to the (0,1) interval.

[0074] When the semantic similarity exceeds a preset verification threshold, the first and second result items are marked as mutually verified result item pairs. The verification threshold is set to 0.65. The basis for setting the verification threshold to 0.65 is as follows: On the manually labeled result item semantic equivalence verification dataset, this dataset contains 500 pairs of intermediate result items retrieved by different semantic expansion search queries. Each pair is labeled as semantically consistent or semantically inconsistent. By calculating the semantic similarity of each pair of result items and traversing all possible thresholds in the range of 0.5 to 0.8, the threshold point that maximizes the F1 value is selected. Statistically, the F1 value reaches its maximum value of 0.87 when the threshold is 0.65. Therefore, the verification threshold is determined to be 0.65.

[0075] The process of marking mutually verified result item pairs is as follows: Iterate through all pairwise combinations of intermediate search result sets. For each pair of intermediate search result sets, pair each result item from the first intermediate search result set with each result item from the second intermediate search result set. Calculate the semantic similarity between each pair of result items. If the semantic similarity exceeds 0.65, record the result item pair as a mutually verified result item pair and store the corresponding semantic similarity value. Optionally, mutual verification can be extended to three or more semantically extended search queries. When the semantic similarity between multiple result items exceeds the verification threshold, these result items are marked as a mutually verified result item group.

[0076] A first fusion weight is assigned to result items that have passed mutual verification, which is higher than that of result items that have failed verification. The first fusion weight is calculated as follows: Each result item is assigned an initial base weight, which is set to 1.0. For each result item, all mutually verified result item pairs in which that result item participates are retrieved, and the semantic similarity between that result item and the other result item is accumulated and added to the initial base weight as the first fusion weight. If a result item has not participated in any mutually verified result item pairs, the first fusion weight remains at 1.0. Optionally, the semantic similarity can be multiplied by a decay coefficient before accumulation, with the decay coefficient set to 0.5, based on the principle that when two result items mutually verify each other, the confidence gain obtained by each result item from the other should not exceed half of the other's own confidence.

[0077] The results from all intermediate search result sets are sorted and deduplicated based on the first fusion weight. The sorting method is as follows: results from all intermediate search result sets are aggregated into a single set and sorted in descending order according to the first fusion weight of each result item. When multiple result items have the same first fusion weight, a secondary sort is performed based on the number of intermediate search result sets from which the result item originates, with result items from more intermediate search result sets appearing first; if the number of intermediate search result sets is also the same, then the result items are sorted according to their generation time.

[0078] The deduplication method is as follows: The sorted list of result items is traversed from beginning to end, maintaining a set of retained result items. For the currently traversed result item, the core entity node identifier corresponding to that result item in the multi-source semantic knowledge graph is extracted. If the core entity node identifier is the same as the core entity node identifier of any result item in the retained result item set, the current result item is a duplicate and is removed; if the core entity node identifier is different, the current result item is added to the retained result item set and retained in the final result list. After traversal, second source information is added to the retained result items. The second source information records which intermediate search result sets the result item came from before deduplication and the list of duplicate result items merged during deduplication. The deduplicated result item list is the fused search result set. Each result item in the fused search result set contains a subgraph reference in the multi-source semantic knowledge graph, a first fusion weight, second source information, and a semantic representation.

[0079] Example 5:

[0080] See Figure 6 After generating the fused search result set, a dynamic feedback reordering step is performed. In practice, user click or browsing behavior data for at least one result item in the fused search result set is received. This click or browsing behavior data is collected by the application client and sent to the search server via a log reporting channel. The behavior data includes the user's identity, the identifier of the clicked or viewed result item, the behavior type (click or browse), and the timestamp of the behavior. The search server extracts the target result item that the user is interested in from the behavior data. The target result item is the result item that the user clicked or browsed for more than a preset browsing time threshold. The preset browsing time threshold is set to 3 seconds, based on the fact that in information retrieval user behavior research, a dwell time exceeding 3 seconds usually indicates that the user has effectively engaged with the content.

[0081] After extracting the target result item, the content relevance between the target result item and other unclicked result items in the fused search result set is calculated. The content relevance is calculated as follows: The core entity node corresponding to the target result item in the multi-source semantic knowledge graph is obtained, and a two-hop subgraph of the target result item is extracted centered on the core entity node; this two-hop subgraph is defined as the target subgraph. Similarly, the core entity node corresponding to the unclicked result item in the multi-source semantic knowledge graph is obtained, and a two-hop subgraph of the unclicked result item is extracted centered on the core entity node; this two-hop subgraph of the unclicked result item is defined as the candidate subgraph. The graph edit distance between the target subgraph and the candidate subgraph is calculated. The calculation method for the graph edit distance is consistent with the cross-validation stage, i.e., the minimum number of operations required to convert the target subgraph into a candidate subgraph through adding, deleting, and modifying nodes and edges. The definition of the operation cost is the same as in the aforementioned cross-validation steps. The reciprocal of the calculated graph edit distance is taken as the content relevance between the target result item and the unclicked result item. Optionally, content relevance can also be obtained directly by calculating the cosine similarity between the result semantic vector of the target result item and the result semantic vector of the unclicked result item. The result semantic vector is already stored in the attributes of the corresponding entity node when the multi-source semantic knowledge graph is constructed.

[0082] Based on content relevance, the ranking position of unclicked result items with a relevance higher than a preset re-ranking threshold in the fused search results set is improved. The preset re-ranking threshold is set to 0.55, based on statistical analysis of the relevance distribution on historical user feedback data. The median of the relevance distribution is used as the initial screening boundary, with the median being 0.54, rounded up to 0.55. The specific operation for improving ranking position is as follows: For each unclicked result item in the fused search results set, if the content relevance between the unclicked result item and the target result item is greater than 0.55, the current ranking position of the unclicked result item is moved forward by a certain number of positions. The number of positions moved is calculated by multiplying the original ranking position by a re-ranking adjustment factor, which is set to 0.2. This factor is chosen based on simulation testing, which shows that this ratio can effectively improve the visibility of related result items without excessively disturbing the overall ranking. If multiple unclicked result items meet the content relevance condition, their positions are moved forward sequentially according to their content relevance from high to low.

[0083] Following the step of generating the fused search result set, a personalized filtering step based on user profiles is also performed. In specific implementation, the identity identifier of the user initiating the initial search request is obtained. This identifier is generated by the authentication module when the user logs into the system and is transmitted to the search server along with the initial search request. Based on the identity identifier, the user's long-term interest vector and short-term behavior vector are extracted from the user profile database. The user profile database is a key-value storage system, where the key is the user's identity identifier and the value is the user profile data structure. The user profile data structure contains long-term interest vector and short-term behavior vector fields. The long-term interest vector is a 256-dimensional real-number vector, generated from the user's historical search, click, favorite, and download behavior data over the past year through behavior sequence modeling. The generation method involves inputting the user's behavior sequence into a Transformer-based user interest modeling model and taking the model's output interest representation vector as the long-term interest vector. The short-term behavior vector is also a 256-dimensional real-number vector, generated from the user's real-time operation behavior data over the past 30 minutes through the same Transformer-based user interest modeling model. The model uses the short-term behavior sequence as input and outputs a short-term behavior vector.

[0084] Each result item in the fusion retrieval result set is mapped to a result semantic vector. The result semantic vector has a dimension of 256. The mapping method is to obtain the core entity node corresponding to the result item from the multi-source semantic knowledge graph, read the node semantic vector stored by the core entity node when constructing the multi-source semantic knowledge graph, and obtain the node semantic vector by using the LightGCN graph neural network to perform graph embedding learning on the multi-source semantic knowledge graph. The output dimension is set to 256.

[0085] The first matching degree between each result semantic vector and the long-term interest vector, and the second matching degree between each result semantic vector and the short-term behavior vector are calculated separately. The first matching degree is defined as the cosine similarity between the result semantic vector and the long-term interest vector, and the second matching degree is defined as the cosine similarity between the result semantic vector and the short-term behavior vector. The cosine similarity is calculated by dividing the dot product of the two vectors by the product of the L2 norms of the two vectors, and the value range is [-1, 1]. In the calculation, negative values ​​are truncated to 0, so that the matching degree range is [0, 1].

[0086] The results in the fused search result set are filtered based on the weighted sum of the first and second matching degrees. The formula for calculating the weighted sum is:

[0087]

[0088] in, Represents the result item The overall personalized matching score; Represents the result item The first matching degree between the resulting semantic vector and the long-term interest vector takes a value of [0,1]. Represents the result item The second matching degree between the resulting semantic vector and the short-term behavior vector takes a value of [0,1]. This is the long-term interest weighting coefficient, with a value of 0.4. The rationale for choosing 0.4 is as follows: In offline experiments, user click-through rate was used as the optimization target. A grid search was performed in the interval [0,1] with a step size of 0.1. Experimental results show that when The click-through rate is highest when the coefficient is 0.4, so the weight coefficient for long-term interest is set to 0.4, and the corresponding weight coefficient for short-term behavior is 0.6.

[0089] After calculating the comprehensive personalized matching score for each result item in the fused search result set, result items with a comprehensive personalized matching score lower than the personalized threshold are removed. The personalized threshold is set to 0.25, based on the following criteria: in the user's historical behavior dataset, the distribution of comprehensive personalized matching scores for positive and negative feedback result items is statistically analyzed. The threshold for determining the distinction is based on maximizing information gain, with the threshold corresponding to maximizing information gain being 0.26, rounded down to 0.25. The removal operation is performed after sorting, retaining only result items with a comprehensive personalized matching score greater than or equal to 0.25 as the final search results displayed to the user.

[0090] Before acquiring the raw data from at least two heterogeneous data sources, a step of constructing a dynamic access channel for multi-source data is also included. In practice, an independent connector is configured for each heterogeneous data source. The connector is a software module, initialized via a configuration file. The configuration file contains the data reading protocol and authentication information corresponding to that heterogeneous data source. The data reading protocol specifies the type of communication protocol followed for establishing connections and transmitting data with the heterogeneous data source; for example, JDBC for relational databases, RESTAPI for web service data sources, and SFTP for file server data sources. The authentication information contains the credentials required to access the heterogeneous data source, such as username and password, API key, OAuth token, or digital certificate.

[0091] The central scheduler polls the status of each connector at a preset time interval. The central scheduler is an independent scheduling service process that internally maintains a connector registry, recording all configured connector objects and their status information. The preset time interval is set to 60 seconds. This 60-second interval is based on the system's requirements for data timeliness and the average data update frequency of heterogeneous data sources, striking a balance between data retrieval latency and system resource overhead. The 60-second polling interval ensures that data updates are captured within an acceptable time window, while preventing overload of heterogeneous data sources due to excessively frequent polling. Connector status includes three types: "Ready," "Busy," and "Error." When the central scheduler detects that any connector is in a "Ready" state, it triggers that connector to perform a data retrieval operation. The connector's data retrieval operation process involves: establishing a connection with the heterogeneous data source using the configured data reading protocol, completing authentication through authentication information, sending a data query request or data download request to the heterogeneous data source, and receiving the raw data stream returned by the heterogeneous data source.

[0092] The connector encapsulates the retrieved raw data into a unified format data message. This unified format data message uses the Protocol Buffers serialization format, and its structure includes a data source identifier field, a data retrieval timestamp field, a raw data content field, and a data format type field. The data source identifier field records the unique identifier of the heterogeneous data source from which the raw data originates; the data retrieval timestamp field records the timestamp at which the connector began retrieving data; the raw data content field stores the complete content of the raw data as a byte array; and the data format type field indicates the format of the raw data, such as JSON, XML, or CSV. The connector pushes the encapsulated data message to a shared data buffer. The shared data buffer is middleware based on a distributed publish-subscribe message queue, such as Apache Kafka or RabbitMQ, with topic names pre-planned based on the data source type and data domain. The data message is pushed to the corresponding topic partition for subsequent steps to retrieve raw data from the shared data buffer. The raw data retrieval step retrieves data messages from the shared data buffer and deserializes them to obtain the raw data content by subscribing to the corresponding topic and consuming the data messages, thus completing the unified access to multi-source heterogeneous data.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A multi-source data intelligent retrieval method based on AI and natural language processing, characterized in that, Includes the following steps: Obtain raw data corresponding to at least two heterogeneous data sources, and perform semantic parsing and entity extraction on the raw data to construct a multi-source semantic knowledge graph; Receive an initial search request input by a user in natural language form, perform intent recognition and semantic expansion on the initial search request, and generate at least two semantically expanded search terms; The at least two semantic expansion search terms are mapped in parallel to the corresponding subgraphs of the multi-source semantic knowledge graph, and semantic matching and path exploration are performed to obtain the intermediate search result set corresponding to each semantic expansion search term. Based on the semantic correlation between the at least two semantically extended search terms, the intermediate search result set is cross-validated and weighted to generate a fused search result set.

2. The multi-source data intelligent retrieval method based on AI and natural language processing according to claim 1, characterized in that, The specific steps for performing semantic parsing and entity extraction on the original data to construct a multi-source semantic knowledge graph are as follows: For each piece of raw data in each heterogeneous data source, dependency parsing in natural language processing is used to identify the subject-verb-object structure and modification relations in the raw data. The subject and object in the subject-verb-object structure are taken as entity nodes, and the predicate is taken as the initial relation edge between entity nodes; The attributes of the entity nodes are populated according to the modification relationship, and the initial weights are assigned to the initial relation edges according to the confidence of the dependency parsing, so as to obtain the source semantic subgraph of each heterogeneous data source. By using preset ontology mapping rules, entity nodes representing the same real-world object in semantic subgraphs from different sources are semantically aligned and merged, thereby integrating all semantic subgraphs from different sources into the multi-source semantic knowledge graph.

3. The multi-source data intelligent retrieval method based on AI and natural language processing according to claim 2, characterized in that, The preset ontology mapping rules include direct mapping rules based on string similarity, similarity mapping rules based on semantic vectors, and equivalence class mapping rules based on a predefined ontology library.

4. The multi-source data intelligent retrieval method based on AI and natural language processing according to claim 1, characterized in that, The specific steps for performing intent recognition and semantic expansion on the initial search request to generate at least two semantically expanded search terms are as follows: The initial search request is segmented and tagged with parts of speech to extract the core verbs and core noun phrases from the initial search request; The core noun phrase is input into a pre-trained semantic vector model, and at least two synonyms or near-synonyms with the core noun phrase whose vector distance to the core noun phrase is less than a preset threshold are retrieved in the semantic vector space. The core verb is combined with each of the synonyms or near-synonyms, and the combined phrases are used as the semantic expansion search terms.

5. The multi-source data intelligent retrieval method based on AI and natural language processing according to claim 1, characterized in that, The specific steps for mapping the at least two semantically expanded search terms in parallel to the corresponding subgraphs of the multi-source semantic knowledge graph, performing semantic matching and path exploration, and obtaining the intermediate search result set corresponding to each semantically expanded search term are as follows: Each phrase in the at least two semantically expanded search terms is split into a search entity and a search relation; for each semantically expanded search term, a set of candidate entity nodes matching the search entity is located in the multi-source semantic knowledge graph; Starting from each node in the candidate entity node set, a breadth-first traversal is performed in the multi-source semantic knowledge graph along the direction indicated by the retrieval relationship, collecting the nodes and edges encountered during the traversal to form the intermediate retrieval result set corresponding to each semantic expansion retrieval expression.

6. The multi-source data intelligent retrieval method based on AI and natural language processing according to claim 5, characterized in that, When performing a breadth-first traversal, the maximum traversal depth is dynamically limited based on the path length between the current node and the target node and the sum of the weights of the edges along the path.

7. The multi-source data intelligent retrieval method based on AI and natural language processing according to claim 5, characterized in that, The specific steps for locating the set of candidate entity nodes that match the retrieved entity in the multi-source semantic knowledge graph are as follows: Calculate the edit distance between the string of the retrieved entity and the name string of each entity node in the multi-source semantic knowledge graph; Calculate the cosine similarity between the semantic vector of the retrieved entity and the semantic vector of each entity node in the multi-source semantic knowledge graph; Entity nodes that simultaneously satisfy the condition that the edit distance is less than a first preset value and the cosine similarity is greater than a second preset value are included in the candidate entity node set.

8. The multi-source data intelligent retrieval method based on AI and natural language processing according to claim 7, characterized in that, When the candidate entity node set is empty, a fuzzy expansion mechanism is triggered to split the search entity into several sub-word units, match them separately, and then merge the matching results as the candidate entity node set.

9. The multi-source data intelligent retrieval method based on AI and natural language processing according to claim 1, characterized in that, Based on the semantic relevance between the at least two semantically expanded search terms, the specific steps for cross-validating and weighted fusion of the intermediate search result set to generate the fused search result set are as follows: Calculate the semantic similarity between the first result item in the first intermediate search result set corresponding to the first semantic expansion search expression and the second result item in the second intermediate search result set corresponding to the second semantic expansion search expression; When the semantic similarity exceeds a preset verification threshold, the first result item and the second result item are marked as a pair of result items that have passed mutual verification; Assign a first fusion weight higher than that of the result items that failed verification to the result items that passed mutual verification, and sort and deduplicate the result items in all intermediate search result sets according to the first fusion weight to generate the fused search result set.

10. The multi-source data intelligent retrieval method based on AI and natural language processing according to claim 9, characterized in that, The specific steps for calculating the semantic similarity between the first result item in the first intermediate search result set corresponding to the first semantically expanded search expression and the second result item in the second intermediate search result set corresponding to the second semantically expanded search expression are as follows: Extract the first subgraph corresponding to the first result item in the multi-source semantic knowledge graph, and the second subgraph corresponding to the second result item in the multi-source semantic knowledge graph; Calculate the graph edit distance between the first subgraph and the second subgraph, where the graph edit distance is the minimum number of operations required to transform the first subgraph into the second subgraph through adding, deleting, and modifying nodes and edges; The reciprocal of the graph edit distance is used as the semantic similarity between the first result item and the second result item.

Images