Intelligent semantic matching distributed knowledge base query method, device and equipment

Abstract

The invention provides an intelligent semantic matching distributed knowledge base query method, apparatus and device. The method comprises the steps of obtaining a query request input by a user; preprocessing the query request to obtain target query data; processing the target query data to obtain key information, intention types and retrieval strategies; according to the target query data, performing similarity query in a vector knowledge base to obtain a preliminary screening result; the vector knowledge base is updated according to a preset rule; according to the key information and the intention type, performing correlation judgment on the preliminary screening result to obtain an intermediate screening result; and filtering and sorting the intermediate screening result according to the retrieval strategy to obtain a target screening result. According to the method, the timeliness, the matching precision, the resource utilization efficiency and the expandability of knowledge query can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge query, in particular to a distributed knowledge base query method, device and equipment for intelligent semantic matching. BACKGROUND Traditional knowledge bases rely on manual or periodic batch updates and cannot reflect dynamic changes in information (such as news, technical documents, etc.) in real time. Knowledge is supplemented by searching external databases, but it relies on pre-defined indexes and cannot be updated in real time. The cleaning, storage and retrieval process of massive data (such as multi-source heterogeneous data) is time-consuming, resulting in system response delay and low data processing efficiency. Existing systems have weak matching capabilities for fuzzy queries or semantically similar but differently expressed content (such as "artificial intelligence" and "AI"), affecting the accuracy of the search, and have insufficient semantic understanding; vector indexes need to be periodically rebuilt and cannot be dynamically updated, and have insufficient matching capabilities for semantic nuances, and cannot handle synonyms, polysemous words or cross-domain expression differences. Static architecture has poor scalability and is difficult to adapt to sudden growth in data volume, resulting in wasted computing resources or performance bottlenecks and low resource utilization. SUMMARY

[0002] The technical problem to be solved by the present application is to provide a distributed knowledge base query method, device and equipment for intelligent semantic matching. The timeliness, matching accuracy, resource utilization efficiency and scalability of knowledge query can be improved.

[0003] To solve the above technical problems, the technical solutions of the present application are as follows: A distributed knowledge base query method for intelligent semantic matching, comprising: obtaining a query request input by a user; preprocessing the query request to obtain target query data; processing the target query data to obtain key information, intent type and retrieval strategy; performing similarity query in a vector knowledge base according to the target query data to obtain a preliminary screening result; the vector knowledge base is updated according to a preset rule; judging the relevance of the preliminary screening result according to the key information and intent type to obtain an intermediate screening result; filtering and sorting the intermediate screening result according to the retrieval strategy to obtain a target screening result.

[0004] Optionally, processing the target query data to obtain key information, intent type and retrieval strategy comprises: extracting information from the target query data to obtain key information; classifying the query intent of the key information to obtain an intent type; According to the key information and the intent type, a retrieval strategy is obtained.

[0005] Optionally, according to the target query data, a similarity query is performed in the vector knowledge base to obtain a preliminary screening result, including: The target query data is converted into a vector representation to obtain a target query vector. According to the similarity between the target query vector and the entry vector in the vector knowledge base, a preliminary screening result is obtained.

[0006] Optionally, the vector knowledge base is updated according to a preset rule, including: Updated knowledge data is obtained at a predetermined time interval; After the updated knowledge data is processed by de-duplication and vectorization, intermediate knowledge data is obtained; The intermediate knowledge data is processed by sharding to obtain target knowledge data, which is stored in a distributed node.

[0007] Optionally, the de-duplication processing includes exact de-duplication, fuzzy de-duplication and semantic de-duplication, wherein the exact de-duplication includes: The updated knowledge data is processed by data cleaning to obtain first knowledge data; The first knowledge data is compared with the vector knowledge base hash index table, and the first knowledge data with the same hash value is deleted to obtain second knowledge data.

[0008] Optionally, the fuzzy de-duplication includes: The second knowledge data is processed by minimum hash signature to obtain third knowledge data; After the third knowledge data is segmented, it is compared with the vector knowledge base paragraph bucket, and the third knowledge data located in the same bucket is deleted to obtain fourth knowledge data.

[0009] Optionally, the semantic de-duplication includes: The fourth knowledge data is processed by vectorization using a semantic embedding model to obtain fifth knowledge data; The fifth knowledge data is compared with the documents in the vector knowledge base for similarity, and the fifth knowledge data with a similarity exceeding a preset threshold is deleted to obtain sixth knowledge data.

[0010] Optionally, according to the key information and the intent type, the preliminary screening result is judged for relevance to obtain an intermediate screening result, including: According to the key information and the intent type, an associated business field is obtained; The preliminary screening result is judged for relevance in the associated business field to obtain an intermediate screening result.

[0011] The embodiment of the present application also provides a distributed knowledge base query device for intelligent semantic matching, comprising: An acquisition module is configured to acquire a query request input by a user. A processing module is configured to pre-process the query request to obtain target query data, process the target query data to obtain key information, an intent type and a search strategy, perform a similarity query in a vector knowledge base according to the target query data to obtain a preliminary screening result, update the vector knowledge base according to a preset rule, perform a relevance judgment on the preliminary screening result according to the key information and the intent type to obtain an intermediate screening result, and filter and sort the intermediate screening result according to the search strategy to obtain a target screening result.

[0012] The embodiment of the present application also provides a computing device, comprising one or more processors and a storage device configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the distributed knowledge base query method for intelligent semantic matching.

[0013] The above technical solution of the present application has at least the following technical effects: The above distributed knowledge base query method for intelligent semantic matching can acquire a query request input by a user, pre-process the query request to obtain target query data, process the target query data to obtain key information, an intent type and a search strategy, perform a similarity query in a vector knowledge base according to the target query data to obtain a preliminary screening result, update the vector knowledge base according to a preset rule, perform a relevance judgment on the preliminary screening result according to the key information and the intent type to obtain an intermediate screening result, and filter and sort the intermediate screening result according to the search strategy to obtain a target screening result, thereby improving the timeliness, matching accuracy, resource utilization efficiency and scalability of knowledge query. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 FIG. 1 is a flowchart of the distributed knowledge base query method for intelligent semantic matching of the present application; Figure 2 FIG. 2 is a schematic diagram of the distributed knowledge base query device for intelligent semantic matching of the present application. DETAILED DESCRIPTION

[0015] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present invention can be more thoroughly understood and the scope of the present invention can be fully conveyed to those skilled in the art.

[0016] As Figure 1 shown, an embodiment of the present invention proposes a distributed knowledge base query method for intelligent semantic matching, including: Step S1, obtaining a query request input by a user; Step S2, preprocessing the query request to obtain target query data; Step S3, processing the target query data to obtain key information, intent type, and retrieval strategy; Step S4, performing a similarity query in the vector knowledge base according to the target query data to obtain a preliminary screening result; the vector knowledge base is updated according to a preset rule; Step S5, performing a relevance evaluation on the preliminary screening result according to the key information and intent type to obtain an intermediate screening result; Step S6, filtering and sorting the intermediate screening result according to the retrieval strategy to obtain a target screening result.

[0017] In this embodiment, as Figure 1 shown, in the distributed knowledge base query method for intelligent semantic matching, first, a query request input by a user is obtained. The query request is usually the text information input by the user in the query box; Then, preprocessing such as word segmentation and stemming is performed on the query request to obtain target query data; the original natural language query request is standardized and structured to facilitate subsequent semantic understanding and information extraction; specifically, a Chinese word segmentation tool (such as jieba) is used to segment the input sentence; for example, when the user inputs "Recommend the motor models suitable for low-wind-speed wind power generation in plateau areas", Word segmentation result: ["Recommend", "Suitable", "Plateau", "Area", "Of", "Low-wind-speed", "Wind power generation", "Motor", "Model"]; Stop words are removed, and function words without actual meaning, such as "of", "already", "is", etc., are removed; After processing: ["Recommend", "Suitable", "Plateau", "Area", "Low-wind-speed", "Wind power generation", "Motor", "Model"]; Stem extraction / Synonym normalization, perform normalization processing on some words, for example, "Recommend" can be unified as "Recommend"; Results: ["recommended", "suitable", "plateau", "region", "low wind speed", "wind power generation", "motor", "model"] Thirdly, the target query data is processed to obtain key information, intent type, and retrieval strategy; based on the target query data, a similarity query is performed in the vector knowledge base to obtain preliminary screening results; based on the key information and intent type, the preliminary screening results are evaluated for relevance to obtain intermediate screening results; Finally, according to the search strategy, the intermediate screening results are filtered and sorted to obtain the target screening results; the search result sorting and filtering methods are adjusted according to the strategy parameters specified in the intent; the specific steps of the search strategy execution are:

[0018] Output a list of candidate knowledge fragments sorted by relevance; Dynamically adjust the search strategy based on the user's query intent. For example, for time-sensitive information queries (such as news updates), the system will prioritize returning the latest relevant content; while for questions that require in-depth analysis, more complex reasoning mechanisms may be invoked. User query analysis: Through natural language processing (NLP) of user input, the intelligent matching engine can parse the user's actual needs or intentions and optimize the search process accordingly. For example, it can identify key elements such as time, place, and people contained in the query to provide more accurate answers. Specifically: [ { "content": "GW-140 / 2500 wind turbines are suitable for low wind speed areas and have been deployed in many locations on the plateau.", "source": "Manufacturer's official website", "relevance_score": 0.92, "matched_entities": ["low wind speed", "high plateau"] }, { "content": "MySE series fans are capable of adapting to high altitudes and can be used in wind speeds as low as 4.5m / s." "source": "Technical White Paper", "relevance_score": 0.89, "matched_entities": ["Highland", "Low Wind Speed"] }, { "content": "EN-141 / 2.2MW machine type is suitable for low wind speed plains and plateau areas, and has been installed in the northwest region.", "source": "Product manual", "relevance_score": 0.85, "matched_entities": ["low wind speed", "plateau"] }, { "content": "《Low wind speed wind power development guidelines》points out that small and medium-sized units suitable for plateau terrain need to be popularized.", "source": "Policy document", "relevance_score": 0.75, "matched_entities": ["low wind speed", "plateau"] } The scheme of the application solves the bottleneck of traditional knowledge base in real-time, efficiency and scalability by integrating dynamic knowledge base management, intelligent semantic matching and distributed computing optimization, and the realization approach mainly includes three-level deduplication algorithm, incremental learning framework and edge computing architecture, which provides an efficient and scalable solution for intelligent question answering, real-time data analysis and other scenarios.

[0019] In an optional embodiment of the application, in step S3, the target query data is processed to obtain key information, intent type and retrieval strategy, including: Step S31, information extraction is performed on the target query data to obtain key information; Step S32, the key information is classified according to the query intent to obtain the intent type; Step S33, according to the key information and intent type, the retrieval strategy is obtained.

[0020] In this embodiment, the key information such as entity and context clues essential to understanding user intent is extracted from the processed target query data; for example, the extracted content is as follows:

[0021] The extracted context clues include that the user hopes to obtain specific product model recommendations, and emphasizes the adaptation to the "low wind speed" environment, and the geographical location "plateau" may involve factors such as climate and altitude.

[0022] The intent classification process, for example, the intent type output by the classification model: technology application type query + product recommendation, confidence score: 0.94 (indicating high confidence).​ Reason: The user explicitly requested a recommendation for a certain device (motor model); Application scenarios involving the field of engineering technology (wind power generation); Specific technical parameters (low wind speed) and geographical conditions (high plateau) affect the recommendation results.

[0023] The search strategy generation process automatically configures the appropriate search strategy based on the identified intent type and key information to ensure the relevance and accuracy of the returned results. For example, the strategy configuration is:

[0024] Output: Structured intent description { "intent_type": "Technology application query + product recommendation", "confidence_score": 0.94, "key_entities": { "location": "plateau", "Field": "Wind Power Generation", "Technical Parameters": "Low Wind Speed", "Device Type": "Motor Model" }, "retrieval_strategy": { "domain_filter": ["wind power generation", "wind power equipment", "motor"], "location_filter": "Plateau", "parameter_filter": { "Wind speed adaptability": "Low wind speed (<5m / s)", "Altitude acclimation": "High altitude" }, "sort_by": "Suitability Rating", "data_sources": [ "Industry Report", "Fan manufacturer's official website", "Project Library", "Scientific Research Paper Database" ], "reasoning_required": true } } In an optional embodiment of the present application, in step S4, similarity query is performed in the vector knowledge base according to the target query data to obtain a preliminary screening result, including: In step S41, the target query data is converted into a vector representation to obtain a target query vector. In step S42, a preliminary screening result is obtained according to the similarity between the target query vector and the entry vector in the vector knowledge base.

[0025] In this embodiment, the content fragments that are relatively close to the user query in semantics are quickly screened out from the knowledge base, and the specific operation steps are as follows, The original query request is converted into a vector representation using a pre-trained Chinese semantic embedding model, for example, using a RoBERTa-wwm-ext-large model, Example input: "Recommend low wind speed wind turbine model suitable for highland area" Output: a 768-dimensional vector representation Q_vec.

[0026] The text in the knowledge base is vectorized, and the knowledge base content may include fan model introduction, wind field case, technical manual, policy document, etc. The same embedding model is used to generate a vector for each text fragment, and the cosine similarity is calculated. For each knowledge base entry vector D_i_vec, the cosine similarity between Q_vec and D_i_vec is calculated: similarity = cos(Q_vec, D_i_vec), Set a threshold (for example, 0.6), or return the top 100 most relevant results; obtain a preliminary screening result: return all entries with a similarity higher than the set threshold; the example screening result may include: "GW-140 / 2500 wind turbine is suitable for low wind speed area", "MySE series fan can adapt to highland environment", "Guiding opinion on low wind speed wind power project" and the like.

[0027] In an optional embodiment of the present application, in step S4, the vector knowledge base is updated according to a preset rule, including: In step S43, updated knowledge data is obtained at a predetermined time interval. In step S44, after the updated knowledge data is processed for deduplication and vectorization, intermediate knowledge data is obtained. In step S45, the intermediate knowledge data is processed for sharding to obtain target knowledge data, which is stored in a distributed node.

[0028] In this embodiment, a timing task (e.g. using cron expression) is set to periodically check whether there is new data in the specified data source, and real-time acquisition of multi-source data (such as news, documents, sensor data) through API, crawler or Internet of Things device. The automatic update process is used to clean, de-duplicate and semantically encode the newly collected data, convert the text into a dense vector (such as CLIP, Sentence-BERT), and update it to the vector database. The entire data processing process is automated using ETL tools (such as Apache NiFi). Incremental learning framework, parameter freezing and adapter layer design: only the adapter layer (AdapterLayers) related to the newly added knowledge is fine-tuned to reduce the computational overhead. Knowledge distillation, distill the fine-tuned model parameters to a lightweight version to support edge deployment.

[0029] In an optional embodiment of the present application, in step S44, the de-duplication processing includes precise de-duplication, fuzzy de-duplication and semantic de-duplication, wherein the precise de-duplication includes: Step S4411, data cleaning is performed on the updated knowledge data to obtain first knowledge data; Step S4412, the first knowledge data is compared with the vector knowledge base hash index table, and the first knowledge data with the same hash value is deleted to obtain second knowledge data.

[0030] In this embodiment, the vector knowledge base performs de-duplication processing on newly acquired knowledge. Precise de-duplication uses a hash algorithm (such as SHA-256) to quickly locate completely duplicated data. Fuzzy de-duplication uses local sensitive hashing to identify semantically similar documents, and combines a large language model to determine whether it is redundant content. Semantic de-duplication uses a semantic embedding model for semantic analysis to remove documents with different expressions but the same meaning.

[0031] The precise de-duplication is realized by comparing the hash values of the new document original text and the hash index table of the documents in the knowledge base. First, data cleaning is performed on the updated knowledge data to remove unit symbols, such as converting "kW·h" to "kwh", and unifying the number format, such as converting "12%" to "0.12"; then, hash calculation and comparison are performed, and the first knowledge data with the same hash value is deleted to obtain second knowledge data; specifically, for example, the following documents exist in the knowledge base: H1: "2025Q1 photovoltaic power generation report: regional same period growth of 12%" H2: "Wind turbine maintenance guide V3.2 (2025 revised edition)" H3: "Energy storage battery temperature abnormality warning threshold setting standard" The newly collected data is: D1: "Regional photovoltaic power generation analysis: Q1 same period +12%, installed capacity expansion" D2: "New Wind Turbine Maintenance Manual (Published May 2025)" D3: "Suggestions on Optimizing Temperature Control Parameters for Lithium-ion Battery Energy Storage Systems" After hash calculation and comparison, it is found that the hashes of D1 and H1 are different, so D1 is retained as the second knowledge data.

[0032] In an optional embodiment of the present invention, in step S44, the fuzzy deduplication includes: Step S4421, performing a minimum hash signature on the second knowledge data to obtain third knowledge data; Step S4422 : After the third knowledge data is segmented, it is bucketed with the vector knowledge base, and the third knowledge data in the same bucket is deleted to obtain fourth knowledge data.

[0033] In this embodiment, fuzzy deduplication is achieved by using a local sensitive hashing method. First, a dictionary containing industry terms such as "photovoltaic", "turbine", "SOC", etc. is constructed, and then a minimum hash signature (200 hash functions) of the second knowledge data is generated. For example, D2 signature = [h1=417, h2=209,..., h 200 =853]; then, the local sensitive hashing method is used for bucket retrieval. If D2 and H2 fall into the same bucket (Jaccard similarity ≈ 0.91) and there is no match with other documents, D2 is deleted from the third knowledge data.

[0034] In an optional embodiment of the present invention, in step S44, the semantic deduplication includes: Step S4431: vectorize the fourth knowledge data using a semantic embedding model to obtain fifth knowledge data; Step S4432: perform similarity comparison between the fifth knowledge data and the documents in the vector knowledge base, delete the fifth knowledge data whose similarity exceeds a preset threshold, and obtain sixth knowledge data.

[0035] In this embodiment, the semantic embedding model is used to vectorize the fourth knowledge data to obtain the fifth knowledge data, and then the cosine similarity between the fifth knowledge data and the documents in the vector knowledge base is calculated. If the cosine similarity exceeds the preset threshold, it is deleted from the fifth knowledge data; specifically, the RoBERTa-wwm-ext-large model is used to generate a vector, and then the cosine similarity between D3 and H3 is calculated, cos(D3, H3)=0.95, which exceeds the preset threshold of 0.85, then it is determined that D3 and H3 have the same semantics, and D3 is deleted.

[0036] In this embodiment, the semantic embedding model runtime internal parameters remain unchanged, and the output layer of the semantic embedding model is deployed in the distributed nodes. The semantic embedding model uses an incremental learning framework, and only the adaptive layer (Adapter Layers) related to the new knowledge is fine-tuned, reducing the computational overhead. For example: [Freeze] RoBERTa-wwm-ext-large backbone parameters [Insert] Adapter layer into Transformer Block middle [Fine-tune] Only train Adapter layers and final output layer (e.g. MLM Head or classification head) Knowledge distillation, distill the fine-tuned model parameters to a lightweight version, support edge deployment.

[0037] [Step 1] Prepare the Teacher model: RoBERTa-wwm-ext-large [Step 2] Build the Student model: TinyBERT [Step 3] Use training data to generate soft label / intermediate layer output by Teacher [Step 4] Let Student learn the output distribution and / or hidden state of Teacher [Step 5] Fine-tune Student on downstream tasks [Step 6] Deploy the Student model Distributed computing optimization improves the scalability and real-time performance of the system through data sharding, edge computing and elastic resource scheduling. Specifically, data sharding improves the parallelism of data processing, enabling the system to handle large-scale data sets. Edge computing reduces data transmission latency and improves real-time processing capabilities. Elastic resource scheduling ensures that the system can still run efficiently under high concurrency while avoiding resource waste. The combination of these technologies provides efficient and scalable solutions in intelligent question answering, real-time data analysis and other scenarios.

[0038] According to the pre-configured data sharding strategy, the knowledge base is sharded by topic or time and stored in distributed nodes (such as HDFS). Data sharding can split large-scale data sets into multiple smaller parts and store them on different computing nodes, improving data processing parallelism and handling large-scale data sets, effectively dealing with sudden increases in data volume.

[0039] Then, the fragmented data is sent to the edge computing node (EdgeNode), which performs preliminary data processing and analysis locally to reduce cloud transmission delay. The advantage of edge computing is that for scenarios with high real-time requirements (such as industrial monitoring), preliminary processing is performed near the data source to reduce data transmission time and bandwidth consumption and reduce system response delay.

[0040] The edge computing node (EdgeNode) sends load information to the elastic resource scheduler (ElasticScheduler), which dynamically adjusts computing resources using the elastic resource scheduling capability of the cloud computing platform (CloudPlatform). The cloud computing platform (CloudPlatform) automatically increases or reduces computing resources according to actual load. The advantage of elastic resource scheduling is that it uses the elastic resource scheduling capability of the cloud computing platform to dynamically adjust computing resources according to current work load, ensuring efficient operation of the system under high concurrency and avoiding resource waste.

[0041] The elastic resource scheduler (ElasticScheduler) provides efficient intelligent question and answer services to the intelligent question and answer system (QASystem). The intelligent question and answer system (QASystem) supports real-time data analysis by the real-time data analysis system (RealTimeAnalysis).

[0042] In an optional embodiment of the present application, in step S5, the relevance of the preliminary screening results is judged according to the key information and the intent type to obtain intermediate screening results, including: In step S51, the associated business field is obtained according to the key information and the intent type. In step S52, the relevance of the preliminary screening results is judged in the associated business field to obtain intermediate screening results.

[0043] In this embodiment, the key information and the intent type are used to improve retrieval accuracy, solve the literal mismatch problem, and realize cross-domain understanding and reordering. The relevance of each candidate segment is judged according to the key information and the intent type. The relevance score of each piece of content (for example: a score between 0 and 1) is output. The example processing logic is as follows:

[0044] Then cross-domain content matching is performed; if the user intent involves multiple domains (such as "geographical climate", "motor engineering", "policy support" in this example), the relevance of these domains is attempted to be understood; for example, if a piece of content mentions "highland area wind resource distribution map", although it does not directly mention motor models, it can assist in determining which motors are more suitable.

[0045] The intelligent matching engine performs preliminary screening through traditional vector similarity (such as cosine similarity) and deep semantic analysis of the text using a large-scale language model. This enables the system to identify documents or information segments that are different in literal terms but related in semantics. Cross-domain content matching, through the semantic understanding ability of the large-scale language model, can find relevance between different domains and topics, thereby improving the accuracy of fuzzy queries and cross-domain content matching.

[0046] The present application solves the bottleneck of traditional knowledge bases in real-time, efficiency and scalability by integrating dynamic knowledge base management, intelligent semantic matching and distributed computing optimization. Through a three-level deduplication algorithm, incremental learning framework and edge computing architecture, it provides an efficient and scalable solution for intelligent question answering, real-time data analysis and other scenarios.

[0047] Compared with the prior art, the advantages of the present application are as follows:

[0048] As shown in Figure 2 The embodiment of the present application also provides a distributed knowledge base query device 20 for intelligent semantic matching, comprising: An acquisition module 21 is configured to acquire a query request input by a user. A processing module 22 is configured to pre-process the query request to obtain target query data, process the target query data to obtain key information, intent type and retrieval strategy, perform similarity query in a vector knowledge base according to the target query data to obtain a preliminary screening result, update the vector knowledge base according to a preset rule, perform relevance evaluation on the preliminary screening result according to the key information and the intent type to obtain an intermediate screening result, and filter and sort the intermediate screening result according to the retrieval strategy to obtain a target screening result.

[0049] Optionally, processing the target query data to obtain key information, intent type and retrieval strategy comprises: Performing information extraction on the target query data to obtain key information. Performing query intent classification on the key information to obtain an intent type. Obtaining a retrieval strategy according to the key information and the intent type.

[0050] Optionally, according to the target query data, a similarity query is performed in the vector knowledge base to obtain a preliminary screening result, including: The target query data is converted into a vector representation to obtain a target query vector; According to the similarity between the target query vector and the entry vector in the vector knowledge base, a preliminary screening result is obtained.

[0051] Optionally, the vector knowledge base is updated according to a preset rule, including: The updated knowledge data is obtained at a predetermined time interval; After the updated knowledge data is processed by de-duplication and vectorization, intermediate knowledge data is obtained; The intermediate knowledge data is processed by sharding to obtain target knowledge data, which is stored in a distributed node.

[0052] Optionally, the de-duplication processing includes exact de-duplication, fuzzy de-duplication and semantic de-duplication, wherein the exact de-duplication includes: The updated knowledge data is processed by data cleaning to obtain first knowledge data; The first knowledge data is compared with the vector knowledge base hash index table, and the first knowledge data with the same hash value is deleted to obtain second knowledge data.

[0053] Optionally, the fuzzy de-duplication includes: The second knowledge data is processed by minimum hash signature to obtain third knowledge data; After the third knowledge data is segmented, it is compared with the vector knowledge base paragraph bucket, and the third knowledge data located in the same bucket is deleted to obtain fourth knowledge data.

[0054] Optionally, the semantic de-duplication includes: The fourth knowledge data is processed by vectorization using a semantic embedding model to obtain fifth knowledge data; The fifth knowledge data is compared with the documents in the vector knowledge base for similarity, and the fifth knowledge data with a similarity exceeding a preset threshold is deleted to obtain sixth knowledge data.

[0055] Optionally, according to the key information and the intent type, the preliminary screening result is judged for relevance to obtain an intermediate screening result, including: According to the key information and the intent type, an associated business field is obtained; The preliminary screening result is judged for relevance in the associated business field to obtain an intermediate screening result.

[0056] It should be noted that all implementation manners in the method embodiments described above are applicable to the device embodiments, and the same technical effects can also be achieved.

[0057] Embodiments of the present application also provide a computing device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the intelligent semantic matching distributed knowledge base query method described in the present application. All implementation manners in the method embodiments described above are applicable to the computing device embodiments, and the same technical effects can also be achieved.

[0058] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0059] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the system, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0060] In the embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented by other ways. For example, the device embodiments described above are only schematic, and the division of units is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0061] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0062] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0063] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various program code storage media.

[0064] In addition, it should be noted that in the device and method of the present application, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalent solutions of the present application. Moreover, the steps of performing the above series of processes can naturally be executed in time sequence according to the order of description, but do not necessarily have to be executed in time sequence. Some steps can be executed in parallel or independently of each other. It can be understood by those skilled in the art that all or any steps or components of the method and device of the present application can be realized in hardware, firmware, software or their combination in any computing device (including processors, storage media, etc.) or network of computing devices, which can be realized by those skilled in the art using their basic programming skills after reading the description of the present application.

[0065] Therefore, the object of the present application can also be realized by running a program or a set of programs on any computing device. The computing device can be a commonly known general-purpose device. Therefore, the object of the present application can also be realized only by providing a program product containing program code for realizing the method or device. That is, such a program product also constitutes the present application, and the storage medium storing such a program product also constitutes the present application. Obviously, the storage medium can be any commonly known storage medium or any storage medium developed in the future. It should be noted that in the device and method of the present application, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalent solutions of the present application. Moreover, the steps of performing the above series of processes can naturally be executed in time sequence according to the order of description, but do not necessarily have to be executed in time sequence. Some steps can be executed in parallel or independently of each other.

[0066] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for distributed knowledge base query with intelligent semantic matching, characterized in that, The method comprises the following steps: obtaining a query request input by a user; preprocessing the query request to obtain target query data; processing the target query data to obtain key information, an intent type and a retrieval strategy; performing a similarity query in a vector knowledge base according to the target query data to obtain a preliminary screening result; the vector knowledge base is updated according to a preset rule; judging the relevance of the preliminary screening result according to the key information and the intent type to obtain an intermediate screening result; filtering and sorting the intermediate screening result according to the retrieval strategy to obtain a target screening result. 2.The intelligent semantic matching distributed knowledge base query method according to claim 1, characterized in that, The processing of the target query data to obtain the key information, the intent type and the retrieval strategy comprises the following steps: extracting information from the target query data to obtain the key information; classifying the query intent of the key information to obtain the intent type; obtaining the retrieval strategy according to the key information and the intent type. 3.The intelligent semantic matching distributed knowledge base query method of claim 1, wherein, The similarity query in the vector knowledge base according to the target query data to obtain the preliminary screening result comprises the following steps: converting the target query data into a vector representation to obtain a target query vector; obtaining the preliminary screening result according to the similarity between the target query vector and an entry vector in the vector knowledge base. 4.The intelligent semantic matching distributed knowledge base query method of claim 1, wherein, The updating of the vector knowledge base according to the preset rule comprises the following steps: obtaining updated knowledge data at a predetermined time interval; obtaining intermediate knowledge data after the updated knowledge data is subjected to a deduplication processing and a vectorization processing; obtaining target knowledge data by performing a sharding processing on the intermediate knowledge data and storing the target knowledge data in distributed nodes.

5. The method of claim 4, wherein, The deduplication processing comprises an exact deduplication, a fuzzy deduplication and a semantic deduplication, wherein the exact deduplication comprises the following steps: performing data cleaning on the updated knowledge data to obtain first knowledge data; comparing the first knowledge data with a hash index table of the vector knowledge base, deleting the first knowledge data with the same hash value to obtain second knowledge data.

6. The method of claim 5, wherein, The fuzzy deduplication comprises the following steps: performing a minimum hash signature on the second knowledge data to obtain third knowledge data; segmenting the third knowledge data and performing a paragraph bucketing on the third knowledge data with the vector knowledge base, deleting the third knowledge data located in the same bucket to obtain fourth knowledge data.

7. The method of claim 6, wherein, The semantic deduplication comprises the following steps: performing a vectorization processing on the fourth knowledge data by using a semantic embedding model to obtain fifth knowledge data; comparing the fifth knowledge data with documents in the vector knowledge base in terms of similarity, deleting the fifth knowledge data with a similarity exceeding a preset threshold to obtain sixth knowledge data.

8. The method of claim 1, wherein, The judging of the relevance of the preliminary screening result according to the key information and the intent type to obtain the intermediate screening result comprises the following steps: obtaining an associated business field according to the key information and the intent type; judging the relevance of the preliminary screening result in the associated business field to obtain the intermediate screening result.

9. A distributed knowledge base query apparatus for intelligent semantic matching, characterized in that, The method comprises the following steps: an obtaining module, configured to obtain a query request input by a user; a processing module, configured to preprocess the query request to obtain target query data; processing the target query data to obtain key information, an intent type and a retrieval strategy; Based on the target query data, a similarity query is performed in a vector knowledge base to obtain a preliminary screening result; the vector knowledge base is updated according to a preset rule; and based on the key information and intent type, a relevance evaluation is performed on the preliminary screening result to obtain an intermediate screening result; According to the search strategy, the intermediate screening results are filtered and sorted to obtain target screening results.

10. A computing device, comprising: include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

Images