Knowledge base construction method and device, equipment and medium

Abstract

The embodiment of the invention provides a knowledge base construction method and device, equipment and a medium, and relates to the field of knowledge management. The method comprises the following steps: receiving a construction demand of a knowledge base and inputting a large model, performing analysis by using the large model to generate a construction instruction, converting the construction instruction into a target keyword entry group which can be retrieved by a software robot, sending the target keyword entry group to the software robot, performing data retrieval by using the software robot, collecting original data and sending the original data to the large model, and receiving original data by using the large model, determining target data for constructing the knowledge base, and finally constructing the knowledge base according to the target data. Through the mode, the large model and the software robot are introduced in the knowledge base construction process, the knowledge base construction efficiency is greatly improved, the knowledge base construction requirement of a user can be fully understood, data are accurately obtained, the knowledge base data obtaining accuracy is greatly improved, the user experience is improved, and the user experience is improved. The knowledge base is accurately and efficiently constructed.

Description

Technical Field

[0001] The present application relates to the field of knowledge management, and in particular to a knowledge base construction method, apparatus, device and medium. Background Art

[0002] Traditional knowledge base construction methods rely primarily on manual labor, requiring manual collection of various paper materials such as books, journal articles, and industry reports, which are then reviewed and screened to extract valuable information. This is then classified, organized, and entered according to established knowledge frameworks. This consumes significant manpower and time, and is extremely inefficient. Semi-automatic knowledge base construction tools based on simple search engine technology, which are used in the knowledge base construction field, only search for documents on user-specified topics in a limited number of commonly used databases or websites. These narrow search scopes lack the ability to accurately screen document content, and the returned results are often filled with irrelevant information, requiring manual secondary screening.

[0003] In recent years, there have been some solutions that combine automated scripts with traditional machine learning models. Automated scripts can crawl web content in batches on the Internet, but the crawling process lacks intelligent interaction and cannot flexibly adjust strategies according to user needs. Traditional machine learning models also find it difficult to understand complex user instructions and cannot handle complex construction tasks such as those involving multi-domain cross-knowledge, specific time interval constraints, and mixed requirements of multiple file properties. As a result, the constructed knowledge base is poorly accurate and its practicality is greatly reduced.

[0004] Therefore, how to construct a knowledge base efficiently and accurately has become an urgent problem to be solved. Summary of the Invention

[0005] Based on the above technical problems, the embodiments of the present application provide a knowledge base construction method, device, equipment and medium, aiming to efficiently and accurately realize the construction of a knowledge base according to user instructions.

[0006] A first aspect of an embodiment of the present application provides a method for constructing a knowledge base, the method comprising:

[0007] receiving construction requirements of the knowledge base and inputting the construction requirements into the big model;

[0008] Utilizing the large model to analyze the construction requirements, generate construction instructions for the knowledge base, and convert the construction instructions into a target keyword entry group that can be retrieved by a software robot, and send the result to the software robot;

[0009] Utilizing the software robot to perform data retrieval based on the target keyword entry group, collect raw data, and send the raw data to the large model;

[0010] Using the large model to receive the raw data sent by the software robot, and determine target data for constructing the knowledge base;

[0011] The knowledge base is constructed according to the target data.

[0012] Optionally, using the large model to parse the construction requirements and generate construction instructions for the knowledge base includes:

[0013] Using the large model to perform semantic analysis on the construction requirements to determine whether the construction requirements contain all necessary information for knowledge base construction;

[0014] When the construction requirements do not include all the necessary information for building the knowledge base, the large model is used to conduct dialogue questions to guide the user to complete the construction requirements;

[0015] The completed construction requirements are verified and corrected using the large model, and ambiguous expressions and contradictory requirements in the construction requirements are corrected to generate the corrected construction instructions.

[0016] Optionally, the construction instructions are converted into a keyword term group that can be retrieved by a software robot, including:

[0017] Using the large model to convert the construction instructions into searchable keywords;

[0018] Using a cross-combination algorithm to arrange and combine the keywords to generate an original keyword entry group for retrieval;

[0019] The original keyword entry group is semantically expanded using the large model to generate the target keyword entry group.

[0020] Optionally, using the software robot to perform data retrieval based on the target keyword term group to collect raw data includes:

[0021] Presetting rules for the software robot, giving priority to selecting original data from authoritative data sources, the authoritative data sources at least including: government websites, corporate websites;

[0022] Using the software robot to enter the target keyword term group into a search engine to obtain multiple data sources containing raw data;

[0023] The software robot is used to download and / or extract text from the raw data in the data source.

[0024] Optionally, determining target data for constructing the knowledge base using the received raw data using the large model includes:

[0025] Using the large model, the raw data is screened according to the construction instructions, and the raw data that does not comply with any construction instruction is eliminated;

[0026] The large model is used to perform semantic duplication checking on the filtered original data, and duplicate data is eliminated to obtain the target data.

[0027] Optionally, using the large model to filter the raw data according to the construction instructions further includes:

[0028] When the construction instruction includes a semantically ambiguous long sentence instruction, using the large model to perform semantic analysis on the long sentence instruction through a pre-trained semantic model;

[0029] The semantic similarity between the original data and the long sentence instruction is calculated, and the original data below the similarity threshold is eliminated.

[0030] Optionally, using the large model to filter the raw data according to the construction instructions further includes:

[0031] When the construction instructions include specific format instructions, the large model is used to filter through a regular expression matching algorithm to eliminate original data that does not meet the specific format. The specific format at least includes: file naming follows a specific format.

[0032] Optionally, after determining target data for constructing the knowledge base using the received raw data using the large model, the method further includes:

[0033] Set multiple classification thresholds at different levels;

[0034] Using the large model, the target data is converted into a plurality of vectors, and similarities between the vectors of each target data are calculated;

[0035] The similarities between the vectors of the respective target data are compared with the multiple classification thresholds, and the respective target data are divided into multiple levels of classification according to the comparison results and stored.

[0036] Optionally, after determining target data for constructing the knowledge base using the received raw data using the large model, the method further includes:

[0037] When adding new data to the knowledge base, recalculating the similarity between the original target data in the knowledge base and the new data vector;

[0038] The data in the knowledge base is reclassified according to the calculation results and stored.

[0039] A second aspect of an embodiment of the present application provides a knowledge base construction device, the device comprising:

[0040] A requirement receiving module, configured to receive the building requirements of the knowledge base and input the building requirements into the large model;

[0041] An instruction generation module, configured to analyze the construction requirements using the large model, generate construction instructions for the knowledge base, convert the construction instructions into a target keyword term group that can be retrieved by the software robot, and send the result to the software robot;

[0042] A raw data retrieval module, configured to utilize the software robot to perform data retrieval based on the target keyword entry group, collect raw data, and send the raw data to the large model;

[0043] a target data determination module, configured to use the large model to receive the raw data sent by the software robot and determine target data for constructing the knowledge base;

[0044] A knowledge base construction module is used to construct the knowledge base according to the target data.

[0045] Optionally, the instruction generation module includes:

[0046] A semantic analysis submodule, configured to perform semantic analysis on the construction requirements using the large model to determine whether the construction requirements contain all necessary information for knowledge base construction;

[0047] A dialogue questioning submodule is used to use the large model to conduct dialogue questions when the construction requirements do not contain all the necessary information for the knowledge base construction, so as to guide the user to complete the construction requirements;

[0048] The verification and correction submodule is used to use the large model to verify and correct the completed construction requirements, correct the ambiguous expressions and contradictory requirements in the construction requirements, and generate the corrected construction instructions.

[0049] Optionally, the instruction generation module includes:

[0050] A keyword generation submodule, configured to convert the construction instructions into searchable keywords using the large model;

[0051] The original keyword entry group generating submodule is used to arrange and combine the keywords using a cross-combination algorithm to generate an original keyword entry group for retrieval;

[0052] The target keyword term group generating submodule is used to semantically expand the original keyword term group using the large model to generate the target keyword term group.

[0053] Optionally, the original data retrieval module includes:

[0054] A rule presetting submodule is used to presetting rules for the software robot, giving priority to selecting original data from authoritative data sources, the authoritative data sources at least including: government websites, corporate websites;

[0055] a data source determination submodule, configured to utilize the software robot to input the target keyword term group into a search engine to obtain a plurality of data sources containing raw data;

[0056] The original data acquisition submodule is used to download and / or extract text from the original data in the data source using the software robot.

[0057] Optionally, the target data determination module includes:

[0058] A data screening submodule, configured to use the large model to screen the raw data according to the construction instructions, and to remove raw data that does not comply with any construction instruction;

[0059] The data duplication checking submodule is used to perform semantic duplication checking on the filtered original data using the large model, and eliminate duplicate data to obtain the target data.

[0060] Optionally, the data screening submodule includes:

[0061] a semantic analysis unit, configured to perform semantic analysis on a long sentence instruction with ambiguous semantics by using the large model and a pre-trained semantic model when the construction instruction contains the long sentence instruction;

[0062] The semantic similarity calculation unit is used to calculate the semantic similarity between the original data and the long sentence instruction, and eliminate the original data below the similarity threshold.

[0063] Optionally, the data screening submodule further includes:

[0064] A specific format screening unit is used to use the large model to screen through a regular expression matching algorithm when the construction instructions contain specific format instructions, and to eliminate the original data that does not meet the specific format. The specific format at least includes: file naming follows a specific format.

[0065] Optionally, the target data determination module includes:

[0066] A classification threshold setting submodule is used to set classification thresholds at multiple different levels;

[0067] A vector similarity calculation submodule is used to convert the target data into multiple vectors using the large model and calculate the similarity between the vectors of each target data;

[0068] The classification storage submodule is used to compare the similarities between the vectors of the various target data with the multiple classification thresholds, and to classify the various target data into multiple levels of classification according to the comparison results, and store them.

[0069] Optionally, the target data determination module further includes:

[0070] A vector similarity calculation submodule, configured to recalculate the similarity between the original target data in the knowledge base and the newly added data vector when new data is added to the knowledge base;

[0071] The classification storage submodule is used to reclassify the data in the knowledge base according to the calculation results and store them.

[0072] A third aspect of an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the knowledge base construction method of the first aspect of the embodiment of the present application is implemented.

[0073] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the knowledge base construction method of the first aspect of the embodiment of the present application is implemented.

[0074] A fifth aspect of an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the knowledge base construction method of the first aspect of the embodiment of the present application.

[0075] Through the knowledge base construction method of the embodiment of the present application, after receiving the user's knowledge base construction requirements, the user requirements are input into the big model. The big model can parse the user's construction requirements and evaluate the completeness of the requirements. Through the question-and-answer mechanism, the user is guided to complete the elements necessary for the construction requirements; then the construction instructions are converted into a target keyword term group for the software robot to obtain data, and sent to the software robot. The software robot searches for data in the search engine based on the target keyword term group, and sends the collected raw data back to the big model; then the big model screens, checks for duplicates, classifies, and stores the raw data, and determines the target data for constructing the knowledge base; finally, the knowledge base required by the user is constructed based on the target data.

[0076] In this application, based on the previous method of constructing a knowledge base, large models and software robots are introduced to replace the original processes that require repetitive manual operations, greatly improving the efficiency of knowledge base construction. At the same time, unlike the previous simple script operations, the combination of large models and software robots can fully understand the user's needs and focus on building a knowledge base, accurately obtain the data required for the knowledge base, greatly improve the accuracy of knowledge base data acquisition, make the data in the knowledge base more relevant, more in line with the user's usage direction, improve the user's usage experience, and realize accurate and efficient knowledge base construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0078] Figure 1 This is a flow chart of a knowledge base construction method proposed in one embodiment of the present application;

[0079] Figure 2 This is a schematic diagram of a knowledge base construction system proposed in one embodiment of the present application;

[0080] Figure 3 This is a structural block diagram of a knowledge base construction device provided in one embodiment of the present application;

[0081] Figure 4 This is a schematic diagram of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0083] In the drawings, the sizes of components, layer thicknesses, or regions may be exaggerated for clarity. Therefore, any implementation of the present disclosure is not necessarily limited to the dimensions shown in the drawings, and the shapes and sizes of components in the drawings do not reflect true proportions. Furthermore, the drawings schematically illustrate idealized examples, and any implementation of the present disclosure is not limited to the shapes or values ​​shown in the drawings.

[0084] In the related technologies, whether it is manual screening or the use of simple automated tools, in the process of building a knowledge base, it is always difficult to accurately locate the data that meets the actual needs of the builder when collecting data. The manual screening method is easily affected by subjective factors and is also prone to errors. Simple automated tools lack the function of semantic understanding, so they cannot accurately obtain data when receiving complex user instructions. In the process of collecting data for knowledge base, due to the classification method based on simple surface features, it is impossible to classify according to the deep semantic information of the data, resulting in incomplete classification or adding data to the wrong category. This not only easily causes confusion of data in the knowledge base during the construction process, but also easily misleads users in the subsequent use of the knowledge base, making it difficult for users to obtain the data they need. Moreover, the existing knowledge base lags behind in the update and iteration of knowledge, and cannot update the content in the knowledge base in a timely manner according to emerging technologies, resulting in poor timeliness of the knowledge base and poor user experience.

[0085] Therefore, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, the embodiment of the present application proposes a knowledge base construction method, which can not only efficiently collect data during the knowledge base construction process, but also accurately understand the user's knowledge base construction needs, provide accurate knowledge base data and data classification, provide strong support for user use, and facilitate the iteration and update of data in the knowledge base.

[0086] Please refer to Figure 1 , Figure 1 This is a flow chart of a knowledge base construction method of this application. Figure 1 As shown, the method may include steps S101 to S105:

[0087] Step S101: receiving the construction requirements of the knowledge base and inputting the construction requirements into the big model;

[0088] Step S102: using the large model to parse the construction requirements, generate construction instructions for the knowledge base, and convert the construction instructions into a target keyword entry group that can be retrieved by the software robot, and send the result to the software robot;

[0089] Step S103: using the software robot to perform data retrieval based on the target keyword entry group, collect raw data, and send the raw data to the large model;

[0090] Step S104: using the large model to receive the original data sent by the software robot, and determining target data for constructing the knowledge base;

[0091] Step S105: construct the knowledge base according to the target data.

[0092] The knowledge base construction method proposed in this application is mainly divided into three parts. The first is the part of obtaining the user's needs for building a knowledge base. The user's knowledge base construction needs are received through an interface connected to the user, and the needs are input into the big model. The user's needs are parsed by the big model. On the basis of understanding the user's needs, the user's needs are converted into the form of keyword term groups that can be understood by the software robot for use by the software robot; the second part is the part of obtaining the data required for the knowledge base. The keyword term group given by the big model is received by the software robot, and searched on the search engine. The search results are downloaded and saved, and sent to the big model as the original data for knowledge base construction. Since the data contained in the original data is redundant, the data needs to be screened and classified by the big model, and the classified data is used as the target data for building the knowledge base; finally, the target data is segmented into files and the segmented knowledge points are vectorized to build the knowledge base required by the user.

[0093] In this way, based on the original knowledge base construction method, the big model and software robot are introduced to work together. Through the semantic understanding, intent recognition, instruction verification and other functions of the big model, an accurate understanding of the user's needs for building a knowledge base is achieved, ensuring that the target data obtained is data that meets user needs. At the same time, through the efficient data collection means of the software robot, relevant data that can be retrieved on the Internet is collected in a timely manner according to the user needs given by the big model, replacing the tedious work of manual search, greatly improving efficiency and accuracy. Finally, secondary verification and classified storage are performed through the big model to ensure that every piece of data entering the knowledge base strictly meets the user's needs, thereby achieving the purpose of efficiently and accurately building a knowledge base.

[0094] Step S101: receiving the construction requirements of the knowledge base, and inputting the construction requirements into the big model.

[0095] In the embodiments of the present application, the knowledge base construction requirements are received by the user through information input at the interactive level. In an optional embodiment, a visual interactive interface can be provided to guide the user to input the knowledge base construction requirements in natural language. Corresponding guidance can be provided before the user submits the requirements, facilitating the user to submit the accurate and complete knowledge base construction requirements. Subsequently, the received user construction requirements are sent to the large model to achieve the parsing of the user construction requirements.

[0096] Step S102: Utilize the large model to parse the construction requirements, generate construction instructions for the knowledge base, and convert the construction instructions into a target keyword term group that can be retrieved by a software robot, and send the result to the software robot.

[0097] In the embodiment of the present application, after receiving the user's construction requirements, the big model first needs to parse them. The construction requirements input by the user may include many aspects such as the subject area, data type, and scale requirements of the knowledge base. The big model needs to conduct in-depth semantic analysis of the user's construction requirements through natural language processing technology to identify the various conditions required for building the knowledge base, such as: the name of the knowledge base, that is, the subject area of ​​the knowledge base this time; the nature of the file, that is, the type of data required to collect data for the knowledge base, such as technical papers, industry standards, practical cases, etc.; time limit, that is, to clarify the timeliness requirements of the collected data, such as published in the past 2 years, published in the past 5 years, etc.; language preference, that is, the language of the data required by the user, such as giving priority to Chinese materials, giving priority to English materials, etc.; format requirements, that is, the format of the file required by the user, such as PDF format, TXT format, etc. The big model identifies the various needs of the user through semantics and converts them into the constraints required for building the knowledge base in the subsequent process, that is, construction instructions.

[0098] After obtaining the build instructions through the user's build requirements, they need to be converted into a form that the software robot can understand so that the software robot can issue search instructions. In this application, each build instruction is processed to form multiple keyword terms, which are sent to the software robot for retrieval in this form to ensure the comprehensiveness of the retrieved data and avoid data omissions.

[0099] Step S103: Utilize the software robot to perform data retrieval according to the target keyword term group, collect raw data, and send the raw data to the large model.

[0100] In an embodiment of the present application, after receiving the keyword term group sent by the large model, the software robot can be activated to start data retrieval, collect original data that meets the keyword term group through various search websites, and download the original data and send it back to the large model for data screening and organization.

[0101] In an optional embodiment, the software robot can be an intelligent RPA robot (Robotic Process Automation). The RPA robot is a technology that simulates human operations through software robots and automatically performs digital tasks with clear rules and high repetition. It has low development difficulty, low cost, fast implementation speed, and low system dependence, and is suitable for repetitive tasks with clear rules. In this application, it is used as a data retrieval method in the process of building a knowledge base, and clear retrieval instructions are issued to it through a large model. Its efficient and simple working characteristics are used to replace the tedious work of manually collecting data, greatly improving work efficiency and improving work accuracy.

[0102] Step S104: using the large model to receive the original data sent by the software robot, and determining target data for constructing the knowledge base.

[0103] In an embodiment of the present application, after the large model receives a large amount of raw data returned by the software robot, it still needs to be processed. Since the software robot uses a keyword term group during the retrieval process, it cannot include all the construction instructions issued by the user. Therefore, it needs to be screened and the data that meets the various construction instructions is retained as the final target data of the knowledge base.

[0104] Step S105: construct the knowledge base according to the target data.

[0105] In the embodiment of the present application, after determining the target data required for the knowledge base, it is also necessary to finally construct a structured knowledge base, and the required method is to further process the collected data, that is, file segmentation and vectorization. First, the target data is downloaded and saved in the form of a file, but the knowledge base needs to cut the content contained in the file into multiple independent simple fragments for easy retrieval and use. Therefore, the target data is subjected to file segmentation processing, and the key paragraphs are extracted according to the coherence and core themes, and combined into simple fragments to ensure that each fragment focuses on an independent knowledge point, which is convenient for the retrieval and application of the content in the knowledge base. In an optional embodiment, data of different content forms can be segmented according to different rules. For example, for data with clear chapter divisions in the article content, such as policies and regulations, it can be parsed based on the article template, and the long article can be split into multiple fragments with clear themes according to clues such as the article title, chapter title, and paragraph format. For data with non-standard text formats, such as market analysis reports in plain text format, natural language processing can be used to extract text summaries and topic models, such as through the LDA (Latent Dirichlet Allocation) model to identify the text topic distribution, thereby achieving file segmentation.

[0106] The segmented text blocks need to be vectorized to obtain multiple text block vectors. Then, tools such as graph databases and vector search engines are used to build the underlying architecture of the knowledge base. A knowledge network is constructed with text block vectors as the base point and knowledge associations as the edges, thereby realizing the construction of a multi-dimensional knowledge base.

[0107] In combination with the above embodiments, in one implementation, the present application further provides a knowledge base construction method, which uses the large model to parse the construction requirements and generate construction instructions for the knowledge base, specifically including the following contents:

[0108] First, the large model is used to perform semantic analysis on the construction requirements to determine whether the construction requirements contain all the necessary information for knowledge base construction.

[0109] In the embodiment of the present application, after receiving the user's construction requirements, the large model needs to perform semantic analysis as described above to identify the construction instructions for the knowledge base. Obtaining construction instructions in this way is also an advantage of the large model, which can more accurately parse the user's natural language into construction instructions that can be used for subsequent data retrieval and data screening. In an optional embodiment, the construction instructions for different knowledge bases can be adjusted accordingly according to the field to which the knowledge base to be constructed belongs and the needs of the user. By adjusting the types of necessary construction instructions obtained by the large model, unnecessary guidance to the user can be avoided, and the generated construction instructions can also be made more accurate.

[0110] Then, when the construction requirements do not include all the necessary information for building the knowledge base, the large model is used to conduct dialogue questions to guide the user to complete the construction requirements.

[0111] In an embodiment of the present application, in order to avoid incomplete information contained in the construction requirements input by the user at one time, the large model will activate the questioning mechanism when it finds that the extracted construction requirements cannot meet all the necessary information. Through multiple rounds of dialogue, inquiries will be asked to the user to guide the user to supplement the construction requirements to meet the needs of knowledge base construction.

[0112] Finally, the completed construction requirements are verified and corrected using the large model, and the ambiguous expressions and contradictory requirements in the construction requirements are corrected to generate the corrected construction instructions.

[0113] In an embodiment of the present application, after the big model obtains the user's construction requirements, it also needs to perform certain verification and correction on the construction requirements. Through the intention recognition and instruction verification functions integrated in the big model, some vague expressions and obviously inconsistent language in the construction requirements input by the user are identified. By verifying and correcting the construction requirements, it is ensured that the final parsed construction instructions are accurate, complete, and executable.

[0114] In conjunction with the above embodiments, in one implementation, the present application further provides a knowledge base construction method, which converts the construction instructions into a keyword term group that can be retrieved by a software robot, specifically including the following contents:

[0115] First, the construction instructions are converted into searchable keywords using the large model.

[0116] In an embodiment of the present application, after parsing the user's construction instructions, it is also necessary to convert the construction instructions into a form that can be retrieved by the software robot. The specific method is to first convert the construction instructions into the form of searchable keywords through the big model. In an optional embodiment, the big model performs a deep semantic understanding of the construction instructions and decomposes the text information in the construction instructions into two dimensions: an object list and an attribute list. The object list represents the core entities or concepts, and the attribute list represents the characteristics, limiting conditions or associated elements of these entities. In this way, the generated keywords are more accurate and easy to retrieve.

[0117] Then, a cross-combination algorithm is used to arrange and combine the keywords to generate an original keyword entry group that can be searched.

[0118] In the embodiment of the present application, in order to avoid a single keyword search content being too large and multiple keyword combinations limiting the search scope, a cross-combination algorithm is selected to arrange and combine the keywords in pairs to generate a keyword term group that is easy to search and contains a certain target range. By adopting this form of keyword term group, the problems of the keyword generation method in related technologies, such as the lack of deep semantic understanding and the single keyword combination, are solved, providing a high-quality query basis for the subsequent software robot to perform data retrieval.

[0119] Finally, the original keyword entry group is semantically expanded using the large model to generate the target keyword entry group.

[0120] In an embodiment of the present application, in order to further enrich the data retrieval scope required by the user, the synonym expansion technology of the large model is used to semantically expand the keywords using a pre-trained language model, and the expanded results are also used as part of the retrieval conditions, thereby enriching the content of the original keyword term group and obtaining the target keyword term group, thereby ensuring the comprehensiveness of the query.

[0121] In an optional embodiment, a keyword weighting algorithm can be used to rank the generated keywords by importance, prioritizing the search for keyword terms consisting of the top-ranked keywords, thereby optimizing data retrieval efficiency. Furthermore, based on the domain knowledge graph, relevant technical terms can be intelligently provided for the domain in which the user wishes to build a knowledge base, thereby improving data retrieval accuracy.

[0122] In conjunction with the above embodiments, in one implementation, the present application further provides a knowledge base construction method, which utilizes the software robot to perform data retrieval based on the target keyword term group and collect original data, specifically including the following contents:

[0123] First, rules are preset for the software robot, and original data from authoritative data sources are preferentially selected. The authoritative data sources include at least: government websites and corporate official websites.

[0124] In an embodiment of the present application, a software robot receives a keyword entry group obtained by parsing a large model to perform data retrieval. To ensure that the retrieved data is more authoritative, rules can be preset for the software robot. Specifically, certain data sources can be set as authoritative data sources during the retrieval process, and data can be obtained preferentially from authoritative data sources, such as government websites that publish policies and regulations in relevant fields, websites of departments at all levels that publish specific implementation details, official corporate websites such as trading platforms and consulting agencies, and online platforms for publishing academic materials. By presetting rules for the software robot, more authoritative data sources are given priority, avoiding the problem of the obtained raw data being too redundant and having low confidence, and more accurately collecting the data required by the user.

[0125] Then, the software robot is used to input the target keyword term group into a search engine to obtain multiple data sources containing raw data.

[0126] In an embodiment of the present application, a software robot uses a preset program to enter the target keyword term group given by the large model into a search engine, retrieves multiple data sources containing raw data, and then enters each data source according to the preset program to obtain the raw data therein. In an optional embodiment, the software robot can purposefully collect raw data from the data source based on the construction instructions parsed by the large model. For example, on a government website, the software robot first selects files published within a certain period of time based on the time range of the construction instructions, and then retrieves the required raw data from the above files based on the target keyword term group. In this way, the data retrieval efficiency is further improved and the reliability of the raw data is improved.

[0127] During the software robot's data retrieval process, some anomalies may occur due to network fluctuations or data source website regulations. In an optional embodiment, when the software robot encounters network fluctuations, page load timeouts, and other issues, it can adjust the request frequency and number of retries according to preset policies. If the software robot encounters problems such as being unable to crack a verification code, it will send a request for help to operations personnel and record the anomaly information. By enriching the software robot's strategies for responding to different situations, the stability of the data collection process can be improved.

[0128] Finally, the software robot is used to download and / or extract text from the raw data in the data source.

[0129] In the embodiment of the present application, after determining the data source of the raw data, the software robot also needs to collect the raw data. Specifically, for data sources that include download links, the software robot can directly download the raw data files in the corresponding format required by the user and monitor the download progress in real time. For data sources that do not include download links, the software robot can use text extraction tools to capture text, charts, and other elements related to the raw data in the data source to ensure the integrity of the raw data. Finally, the collected raw data files or the raw data text extracted through text extraction are transferred to the large model for further screening and classification.

[0130] In combination with the above embodiments, in one implementation, the present application further provides a knowledge base construction method, which uses the large model to determine target data for constructing the knowledge base through the received raw data, specifically including the following contents:

[0131] First, the large model is used to screen the raw data according to the construction instructions, and the raw data that does not comply with any construction instruction is eliminated.

[0132] In the embodiment of the present application, after receiving a large amount of raw data obtained by the software robot, the large model needs to further screen it, retain the data that meets the needs of the user, and ensure that the data in the knowledge base is of high quality and high relevance. Specifically, the data screening must first remove the garbled information, advertising information, and other data that are obviously unreadable or do not meet the needs of the user; then, the large model needs to screen the collected raw data for a second time. Since the raw data is collected based on the keyword term group, it may only contain a part of the construction instructions. Therefore, the large model needs to screen out the data in the raw data that meets all the construction instructions proposed by the user according to the user's construction instructions, as the data to be stored in the knowledge base, and for other data that cannot meet the construction instructions, it needs to be proposed to ensure that every file entering the knowledge base strictly conforms to the construction instructions given by the user.

[0133] Then, the large model is used to perform semantic duplication check on the filtered original data, and duplicate data is eliminated to obtain the target data.

[0134] In an embodiment of the present application, after the data that meets the construction instructions is obtained through secondary screening, it is also necessary to check the data for duplicates, and perform semantic duplication checking on the filtered original data through the large model to avoid the redundancy of the data stored in the knowledge base due to the software robot obtaining similar original data from different data sources. In an optional embodiment, the large model can use a semantic duplication checking algorithm based on deep learning to build a fingerprint model of text semantics, convert each piece of data into a unique semantic fingerprint, and quickly identify duplicate content by comparing fingerprints. In this way, not only the efficiency of duplication checking is improved, but also data with different expressions but similar semantics can be accurately screened out to avoid data redundancy in the knowledge base.

[0135] In combination with the above embodiments, in one implementation, the present application further provides a knowledge base construction method, which utilizes the large model and filters the raw data according to the construction instructions, and specifically includes the following contents:

[0136] First, when the construction instruction includes a semantically ambiguous long sentence instruction, the large model is used to perform semantic analysis on the long sentence instruction through a pre-trained semantic model.

[0137] In the embodiment of this application, during the secondary screening of raw data described above, there may be long, semantically ambiguous instructions in the construction instructions, such as "focusing on distributed energy trading microgrid practices and feasible solutions." These long instructions are semantically ambiguous and difficult to define as screening conditions using conventional rules. Therefore, it is necessary to perform semantic analysis on the long instructions using a semantic model pre-trained by a large model and use a semantic similarity algorithm to deeply mine the data. In this way, the understanding of user construction requirements is improved, and a knowledge base that meets user needs can be more appropriately established.

[0138] In an optional embodiment, long sentence instructions can also be analyzed through a machine learning classification strategy. Through training with a large amount of labeled training data, the information that matches the semantics of the long sentence is classified as positive, and the others are classified as negative. The machine learning model for classification is trained through algorithms such as Naive Bayes and Support Vector Machine, and the trained model is used as a tool for screening long sentence instructions.

[0139] Then, the semantic similarity between the original data and the long sentence instruction is calculated, and the original data below the similarity threshold is eliminated.

[0140] In the embodiment of the present application, the large model determines whether the filtered original data meets the requirements of the long sentence instruction by calculating the semantic similarity between the original data to be filtered and the long sentence instruction after semantic analysis. Optionally, a reasonable similarity threshold can be set based on the accuracy requirements of the knowledge base and the data filtering experience of previous knowledge base construction. For original data with semantic similarity higher than the similarity threshold, it is considered that it meets the requirements of the long sentence instruction and is retained, while the original data below the similarity threshold will be eliminated. In this way, it is ensured that the filtered data deeply fits the user's construction needs at the semantic level, and the accuracy of the data in the knowledge base is improved.

[0141] In combination with the above embodiments, in one implementation, the present application further provides a knowledge base construction method, which utilizes the large model and filters the raw data according to the construction instructions, and specifically includes the following contents:

[0142] When the construction instructions include specific format instructions, the large model is used to filter through a regular expression matching algorithm to eliminate original data that does not meet the specific format. The specific format at least includes: file naming follows a specific format.

[0143] In an embodiment of the present application, the knowledge base construction instructions proposed by the user may also include some specific instructions that require the format, which may appear in some policy documents with specific naming specifications. For example, it is stipulated that file naming should follow the format of "ET_region code_YYYYMMDD_serial number". In the screening process, it is necessary to screen out data that meets the naming specifications. In this application, it is proposed to use a large model to screen the data with such specific format requirements through a regular expression matching algorithm, and to construct an adaptive regular expression pattern to implement precise screening. For example, for data in a specific naming format as described above, the title of the data can be screened through the regular expression: "r'ET_\d{3}_\d{8}_\d{5}'".

[0144] In combination with the above embodiments, in one implementation, the present application further provides a knowledge base construction method, which, after determining target data for constructing the knowledge base using the received raw data using the large model, specifically includes the following:

[0145] First, multiple classification thresholds at different levels are set.

[0146] In the embodiment of the present application, after the target data is obtained by screening and checking the original data using the large model, the target data needs to be stored. In order to ensure that the structure of the knowledge base is clear and easy to retrieve and manage, the target data needs to be classified according to certain rules before being stored. This application proposes to manage the target data in a vectorized manner, and then by setting classification thresholds at different levels and calculating which level threshold it is at, a large amount of target data can be classified into different categories to form a storage architecture with clear hierarchical divisions.

[0147] Then, the target data is converted into a plurality of vectors using the large model, and the similarity between the vectors of each target data is calculated.

[0148] In an embodiment of the present application, the target data is vectorized by a large model, and the similarity between the vectors is calculated to determine whether the data is related and whether it can be divided into the same category. Specifically, in an optional embodiment, a bag-of-words model BoW (Bag of Words) can be used to build a vocabulary, and the different words appearing in the data are counted to generate corresponding bag-of-words vectors to represent the number of times each word in the data appears in the text, and then the inverse document frequency IDF (Inverse Document Frequency) of each word is calculated to obtain the vector of the target data; a large number of word vector models can also be obtained through the word embedding model Word2Vec, and then the vectors of the words contained in the target data are calculated by summing, averaging, etc. to obtain the vector of the target data.

[0149] After vectorizing each target data, it is also necessary to calculate the similarity between the vectors of each data as a classification criterion. Specifically, in an optional embodiment, the cosine similarity between two data vectors can be calculated. The closer the calculation result is to 1, the more similar the semantics between the two data are. The Euclidean distance between the two data vectors can also be calculated to make a judgment. The smaller the Euclidean distance between the two vectors, the closer the two vectors are, and the higher the text similarity between the corresponding two data is.

[0150] The similarities between the vectors of the respective target data are compared with the multiple classification thresholds, and the respective target data are divided into multiple levels of classification according to the comparison results and stored.

[0151] In the embodiment of the present application, after calculating and obtaining the similarity between each target data, it is classified according to the calculation result, and the calculated similarity is compared with the pre-set classification threshold. If it is greater than or equal to the classification threshold, it means that the two data belong to the same category, and they are classified into one category. Specifically, in order to achieve classification results at different levels, the similarity between the target data can be compared with the maximum classification threshold first to obtain the first classification result. The target data is divided into multiple categories with the smallest range, and the data similarity in each category is the highest; then a second classification is performed, and the similarity between data of different categories is compared with the second highest classification threshold, and the two categories of data with the result not less than the second highest classification threshold are aggregated into a large category, thereby obtaining a second classification result with a larger range; and so on, until the similarity between all data is less than the minimum classification threshold, that is, all target data are aggregated into the same large category, thereby forming a hierarchical classification from basic to high-order. In this way, the data stored in the knowledge base is rigorous in structure and clear in context, which is convenient for data storage and retrieval.

[0152] In combination with the above embodiments, in one implementation, the present application further provides a knowledge base construction method, which, after determining target data for constructing the knowledge base using the received raw data using the large model, specifically includes the following:

[0153] First, when new data is added to the knowledge base, the similarity between the original target data in the knowledge base and the new data vector is recalculated; then, the data in the knowledge base is reclassified according to the calculation result and stored.

[0154] In an embodiment of the present application, a method for updating a constructed knowledge base is also proposed. In the case of adding new data to the knowledge base, the present application proposes to reclassify and store the data originally stored in the knowledge base and the newly added data. The purpose is to avoid the situation where the original classification method may be imperfect or unsuitable after the new data is added. By recalculating the similarity between the newly added data vector and the vector of the original target data, and then reclassifying and storing the data in the knowledge base based on the calculation results, it is possible to optimize and adjust the unreasonable classification in a timely manner to ensure that the structure of the knowledge base is reasonable and clear.

[0155] In an optional embodiment, software robots can monitor relevant data sources for updates to the knowledge base. After the knowledge base is built, if relevant knowledge sources are updated with data that meets the user's construction requirements, the software robots can promptly collect and add it to the knowledge base. For example, newly released policies and regulations on government websites can be promptly collected and added to the knowledge base. This approach shortens the data update cycle in the knowledge base and avoids the situation where slow manual updates cause knowledge lag in the knowledge base.

[0156] In another optional embodiment, the present application also proposes a method for updating the knowledge base based on user feedback, collecting user feedback information on data retrieval in the knowledge base through a large model, and timely optimizing the data classification strategy based on low user satisfaction in the feedback information or users’ feedback on outdated data, calling software robots to collect the latest data, so as to meet the user’s usage needs and ensure the update speed of data in the knowledge base.

[0157] This application also proposes an optional knowledge base automatic construction system based on RPA software robots and large models, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a knowledge base construction system provided by an embodiment of the present application. As shown in the figure, first of all, for the user interaction layer, it is mainly used to realize the human-computer interaction between the user and the knowledge base construction system, specifically including providing a visual interactive interface, guiding the user to input the knowledge base construction requirements in natural language, and preliminarily verifying the received construction requirements to ensure smooth and readable, and then sending the construction requirements to the big model. On the other hand, after the big model parses the construction requirements, the user interaction layer is also responsible for feeding back the parsing results of the big model to the user, including guiding the user to supplement the construction instructions through pre-set friendly inquiry words when the user needs to supplement the construction instructions.

[0158] Then, for the big model intelligent analysis layer, the big model is mainly used to parse the user's construction requirements to generate construction instructions, including the instruction parsing part of the construction requirements, identifying construction instructions such as knowledge base name, file nature, time requirements, etc.; and when the construction instructions are incomplete, interactive dialogue is generated according to preset rules and sent to the user interaction layer to guide the user to complete the construction information; and specific requirements in the construction requirements are extracted, such as policies for specific areas, instructions related to specific equipment models, etc.; finally, it also includes generating keywords for retrieval and combining keywords into keyword term groups.

[0159] Next, the RPA execution layer, which performs data retrieval, includes the environment configuration and initialization phase for the work sign-in process. The resource allocation unit determines the CPU, memory, and other computing resources allocated to the RPA software robot, and the parameter initialization unit configures various working parameters for the data retrieval, such as the timeout limit and number of retries for accessing a specific URL. It also includes a web navigation and search phase, in which the RPA software robot uses the browser control unit to open a search engine or a relevant URL, enters the keyword entry group provided by the large model through the keyword input unit, and determines the data source containing the original data in the search results based on the construction instructions through the filter option application unit. It also includes a data extraction and download phase, in which the RPA software robot uses the text extraction unit and RPA's text extraction tools to identify text information that cannot be directly downloaded as original data, and directly downloads the downloadable original data from the data source through the file download unit. It also includes a progress monitoring unit that monitors the download progress of the original data to ensure that the original data is fully acquired. It also includes exception handling and logging, which includes a verification code processing unit, which uses image recognition technology to crack the verification code that may exist in the data source and seek help from operation and maintenance personnel when the cracking fails, and an anomaly detection unit, which is used to monitor abnormal problems during the operation of the RPA software robot, including data source loading timeout, original data download link failure, etc. In order to record these abnormal problems, this part also includes a logging feedback unit, which is used to record the abnormal problems in the work log file of the RPA software robot for inspection by operation and maintenance personnel.

[0160] After completing the data retrieval work, it also includes the data processing layer that uses large models for screening and classification, including a preliminary screening part, specifically including a garbled code repair unit that identifies whether the original data has garbled characters or other unreadable problems, and a rule matching unit that performs preliminary screening of the original data according to the construction instructions, and a format verification unit that screens by detecting the format of the original data file, and a machine learning classification unit and a semantic similarity unit that further screens according to semantic conditions, as well as a regular expression matching unit that screens for specific requirements. After completing the preliminary screening of the original data, the data processing layer also includes a duplicate checking and verification part for data duplication, including a semantic duplication checking unit that identifies duplicate data through a deep learning semantic duplication checking algorithm, and a large model verification unit that performs secondary verification through a large model to ensure the accuracy of the file, and again identifies whether the data after the duplicate check meets the requirements of the user's construction instructions.

[0161] Finally, after determining the target data in the knowledge base, the knowledge base construction layer includes an intelligent classification part, which is used to divide the target data into multiple categories in a regular and hierarchical manner through a hierarchical clustering classification unit that vectorizes the data and calculates the similarity between vectors. It also includes a dynamic classification adjustment unit in the operation and maintenance of the knowledge base in the later stage, after adding new data, which can adjust the data classification structure in the knowledge base to ensure the timeliness of the data in the knowledge base. It also includes a knowledge base construction and storage part, in which the target data is divided into knowledge fragments through file segmentation and vectorization units, and the knowledge fragments are converted into vector representations, so that users can locate more accurately when performing knowledge retrieval. There is also a database construction unit that uses a graph database or vector search engine to build the underlying architecture of the knowledge base, and uses the vectorized knowledge fragments as nodes and the relationships between knowledge fragments as edges to build a knowledge network, realizing efficient storage and intelligent retrieval recommendations based on the mutual relationship between knowledge.

[0162] Based on the same design concept, an embodiment of the present application provides a knowledge base construction device. Figure 3 , Figure 3 This is a structural block diagram of a knowledge base construction device provided by an embodiment of the present application. Figure 3 As shown, the device includes:

[0163] A requirement receiving module, configured to receive the building requirements of the knowledge base and input the building requirements into the large model;

[0164] An instruction generation module, configured to analyze the construction requirements using the large model, generate construction instructions for the knowledge base, convert the construction instructions into a target keyword term group that can be retrieved by the software robot, and send the result to the software robot;

[0165] A raw data retrieval module, configured to utilize the software robot to perform data retrieval based on the target keyword entry group, collect raw data, and send the raw data to the large model;

[0166] a target data determination module, configured to use the large model to receive the raw data sent by the software robot and determine target data for constructing the knowledge base;

[0167] A knowledge base construction module is used to construct the knowledge base according to the target data.

[0168] Optionally, the instruction generation module includes:

[0169] A semantic analysis submodule, configured to perform semantic analysis on the construction requirements using the large model to determine whether the construction requirements contain all necessary information for knowledge base construction;

[0170] A dialogue questioning submodule is used to use the large model to conduct dialogue questions when the construction requirements do not contain all the necessary information for the knowledge base construction, so as to guide the user to complete the construction requirements;

[0171] The verification and correction submodule is used to use the large model to verify and correct the completed construction requirements, correct the ambiguous expressions and contradictory requirements in the construction requirements, and generate the corrected construction instructions.

[0172] Optionally, the instruction generation module includes:

[0173] A keyword generation submodule, configured to convert the construction instructions into searchable keywords using the large model;

[0174] The original keyword entry group generating submodule is used to arrange and combine the keywords using a cross-combination algorithm to generate an original keyword entry group for retrieval;

[0175] The target keyword term group generating submodule is used to semantically expand the original keyword term group using the large model to generate the target keyword term group.

[0176] Optionally, the original data retrieval module includes:

[0177] A rule presetting submodule is used to presetting rules for the software robot, giving priority to selecting original data from authoritative data sources, the authoritative data sources at least including: government websites, corporate websites;

[0178] a data source determination submodule, configured to utilize the software robot to input the target keyword term group into a search engine to obtain a plurality of data sources containing raw data;

[0179] The original data acquisition submodule is used to download and / or extract text from the original data in the data source using the software robot.

[0180] Optionally, the target data determination module includes:

[0181] A data screening submodule, configured to use the large model to screen the raw data according to the construction instructions, and to remove raw data that does not comply with any construction instruction;

[0182] The data duplication checking submodule is used to perform semantic duplication checking on the filtered original data using the large model, and eliminate duplicate data to obtain the target data.

[0183] Optionally, the data screening submodule includes:

[0184] a semantic analysis unit, configured to perform semantic analysis on a long sentence instruction with ambiguous semantics by using the large model and a pre-trained semantic model when the construction instruction contains the long sentence instruction;

[0185] The semantic similarity calculation unit is used to calculate the semantic similarity between the original data and the long sentence instruction, and eliminate the original data below the similarity threshold.

[0186] Optionally, the data screening submodule further includes:

[0187] A specific format screening unit is used to use the large model to screen through a regular expression matching algorithm when the construction instructions contain specific format instructions, and to eliminate the original data that does not meet the specific format. The specific format at least includes: file naming follows a specific format.

[0188] Optionally, the target data determination module includes:

[0189] A classification threshold setting submodule is used to set classification thresholds at multiple different levels;

[0190] A vector similarity calculation submodule is used to convert the target data into multiple vectors using the large model and calculate the similarity between the vectors of each target data;

[0191] The classification storage submodule is used to compare the similarities between the vectors of the various target data with the multiple classification thresholds, and to classify the various target data into multiple levels of classification according to the comparison results, and store them.

[0192] Optionally, the target data determination module further includes:

[0193] A vector similarity calculation submodule, configured to recalculate the similarity between the original target data in the knowledge base and the newly added data vector when new data is added to the knowledge base;

[0194] The classification storage submodule is used to reclassify the data in the knowledge base according to the calculation results and store them.

[0195] Based on the same design concept, another embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in the knowledge base construction method described in any of the above embodiments of the present application.

[0196] Based on the same design concept, another embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the knowledge base construction method described in any of the above embodiments of the present application.

[0197] Based on the same design concept, another embodiment of the present application provides an electronic device, such as Figure 4 shown. Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present application. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of the knowledge base construction method described in any of the above embodiments of the present application.

[0198] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0199] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0200] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0201] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0204] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0205] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0206] The above is a detailed introduction to the knowledge base construction method, device, equipment and medium provided by this application. Specific examples are used in this article to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core ideas. At the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A knowledge base construction method, characterized in that: include: receiving construction requirements of the knowledge base and inputting the construction requirements into the big model; Utilizing the large model to analyze the construction requirements, generate construction instructions for the knowledge base, and convert the construction instructions into a target keyword entry group that can be retrieved by a software robot, and send the result to the software robot; Utilizing the software robot to perform data retrieval based on the target keyword entry group, collect raw data, and send the raw data to the large model; Using the large model to receive the raw data sent by the software robot, and determine target data for constructing the knowledge base; The knowledge base is constructed according to the target data.

2. The knowledge base construction method according to claim 1, characterized in that: Utilizing the large model to parse the construction requirements and generate construction instructions for the knowledge base includes: Using the large model to perform semantic analysis on the construction requirements to determine whether the construction requirements contain all necessary information for knowledge base construction; When the construction requirements do not include all the necessary information for building the knowledge base, the large model is used to conduct dialogue questions to guide the user to complete the construction requirements; The completed construction requirements are verified and corrected using the large model, and ambiguous expressions and contradictory requirements in the construction requirements are corrected to generate the corrected construction instructions.

3. The knowledge base construction method according to claim 1, characterized in that: The construction instructions are converted into a keyword term group that can be retrieved by a software robot, including: Using the large model to convert the construction instructions into searchable keywords; Using a cross-combination algorithm to arrange and combine the keywords to generate an original keyword entry group for retrieval; The original keyword entry group is semantically expanded using the large model to generate the target keyword entry group.

4. The knowledge base construction method according to claim 1, characterized in that: Utilizing the software robot to perform data retrieval based on the target keyword entry group and collect raw data includes: Presetting rules for the software robot, giving priority to selecting original data from authoritative data sources, the authoritative data sources at least including: government websites, corporate websites; Using the software robot to enter the target keyword term group into a search engine to obtain multiple data sources containing raw data; The software robot is used to download and / or extract text from the raw data in the data source.

5. The knowledge base construction method according to claim 1, characterized in that: Determining target data for constructing the knowledge base using the large model based on the received raw data includes: Using the large model, the raw data is screened according to the construction instructions, and the raw data that does not comply with any construction instruction is eliminated; The large model is used to perform semantic duplication checking on the filtered original data, and duplicate data is eliminated to obtain the target data.

6. The knowledge base construction method according to claim 5, characterized in that: Using the large model and screening the raw data according to the construction instructions also includes: When the construction instruction includes a semantically ambiguous long sentence instruction, using the large model to perform semantic analysis on the long sentence instruction through a pre-trained semantic model; The semantic similarity between the original data and the long sentence instruction is calculated, and the original data below the similarity threshold is eliminated.

7. The knowledge base construction method according to claim 5, characterized in that: Using the large model and screening the raw data according to the construction instructions also includes: When the construction instructions include specific format instructions, the large model is used to filter through a regular expression matching algorithm to eliminate original data that does not meet the specific format. The specific format at least includes: file naming follows a specific format.

8. The knowledge base construction method according to claim 1, characterized in that: After determining target data for constructing the knowledge base using the large model through the received raw data, the method further includes: Set multiple classification thresholds at different levels; Using the large model, the target data is converted into a plurality of vectors, and similarities between the vectors of each target data are calculated; The similarities between the vectors of the respective target data are compared with the multiple classification thresholds, and the respective target data are divided into multiple levels of classification according to the comparison results and stored.

9. The knowledge base construction method according to claim 8, characterized in that: include: When adding new data to the knowledge base, recalculating the similarity between the original target data in the knowledge base and the new data vector; The data in the knowledge base is reclassified according to the calculation results and stored.

10. A knowledge base construction device, characterized in that: The device comprises: A requirement receiving module, configured to receive the building requirements of the knowledge base and input the building requirements into the large model; An instruction generation module, configured to analyze the construction requirements using the large model, generate construction instructions for the knowledge base, convert the construction instructions into a target keyword term group that can be retrieved by the software robot, and send the result to the software robot; A raw data retrieval module, configured to utilize the software robot to perform data retrieval based on the target keyword entry group, collect raw data, and send the raw data to the large model; a target data determination module, configured to use the large model to receive the raw data sent by the software robot and determine target data for constructing the knowledge base; A knowledge base construction module is used to construct the knowledge base according to the target data.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by the processor, the knowledge base construction method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the knowledge base construction method according to any one of claims 1 to 9 is implemented.

Images