Method and system for constructing a guarantee knowledge library based on LLM and RAG technologies

Abstract

The application provides a guarantee knowledge library construction method and system based on LLM and RAG technology, and the method comprises the following steps: S1, collecting guarantee knowledge data files and importing them into a retrieval enhancement generation RAG system to form a preliminary version of a knowledge base; S2, loading and analyzing the guarantee knowledge data files in the preliminary version of the knowledge base, storing the parsed text content in the memory to form complete knowledge text data; S3, using an optimized word segmentation strategy to perform word segmentation on the knowledge text data to obtain a plurality of large text blocks and corresponding small text blocks; S4, determining and using an optimal embedding model to perform vector embedding processing on the knowledge blocks composed of the large text blocks and the corresponding small text blocks, and storing the converted vector representation in a vector database to form a guarantee knowledge library. The application can construct a guarantee knowledge library based on LLM and RAG technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language models (LLM) and retrieval augmented generation (RAG), and particularly relates to a method and system for constructing a guarantee knowledge library based on LLM and RAG technology. BACKGROUND

[0002] The guarantee industry, as an important part of the financial system, plays a crucial role in economic activities. In recent years, with the continuous expansion of the financial market and the increase in transaction complexity, the guarantee business faces many challenges, such as increased risk management difficulty, low efficiency of business processes, and the need to improve customer service quality.

[0003] To address these challenges, the industry actively introduces various digital technologies. Traditional guarantee business relies on manual experience and simple information systems for operation, and there are obvious shortcomings in knowledge management. With the application of big data technology, guarantee agencies began to collect and analyze a large amount of business data, establish a guarantee knowledge library, and use the constructed knowledge library for subsequent applications such as guarantee knowledge automatic question answering.

[0004] However, if the data in the guarantee knowledge library contains errors or inaccurate information, the generated results will be biased, and even if a large number of optimizations are performed during the retrieval stage, it is difficult to effectively solve this problem. Moreover, there is information loss in the data vectorization process, and the complexity of text data makes it difficult for limited vectors to fully express all details and features, affecting the accuracy of subsequent document retrieval. Therefore, there is an urgent need for an effective method for constructing a guarantee knowledge library. SUMMARY

[0005] To solve the technical problems existing in the prior art, the present application provides a method and system for constructing a guarantee knowledge library based on LLM and RAG technology, and the technical solution is as follows:

[0006] On the one hand, a method for constructing a guarantee knowledge library based on LLM and RAG technology is provided, which includes:

[0007] S1, collect guarantee knowledge material files and import them into a retrieval augmented generation (RAG) system to form a preliminary knowledge base;

[0008] S2, load and parse the guarantee knowledge material files in the preliminary knowledge base, and store the parsed text content in the memory to form complete knowledge text data;

[0009] S3, use an optimized segmentation strategy to segment the knowledge text data to obtain multiple large text blocks and corresponding small text blocks;

[0010] S4, determine and use the optimal embedding model, and perform vector embedding processing on the knowledge block composed of the large text block and the corresponding small text block, store the converted vector representation in the vector database to form the guaranteed knowledge library.

[0011] Optionally, the S3 specifically includes:

[0012] perform word segmentation on the knowledge text using a word segmentation model based on a pre-trained BERT architecture, and adopt a formula according to the logical structure and semantic integrity of the knowledge content.

[0013] ;

[0014] calculate the block quality score determine the division standard of the large text block, and obtain a plurality of large text blocks with reasonable semantic units;

[0015] wherein, represents the semantic coherence aggregation weight, represents the position dispersion compensation coefficient, represents the punctuation density inhibition coefficient, represents the semantic similarity average of the th adjacent block, represents the starting position index of the current candidate block, represents the reference position index of the th semantic anchor point, represents the number of punctuation symbols in the current block, represents the total number of characters in the current block, represents the number of adjacent blocks.

[0016] After the large text block is divided, the length of each large text block is judged, if the size of the large text block exceeds the pre-set text block length threshold, a length-based word segmentation method is used to perform secondary division on the large text block, to subdivide it into a plurality of small text blocks meeting the length requirement, and an association relationship between the small text blocks and the large text block is established, if the length of the large text block is less than or equal to the pre-set text block length threshold, the large text block is directly saved as a complete unit.

[0017] Optionally, the establishment of the association relationship between the small text block and the large text block specifically includes:

[0018] add a "parent_block_id" field in the data structure to record the large text block information to which the small text block belongs.

[0019] Optionally, the length-based word segmentation method performs automatic word segmentation processing by setting the word segmentation length and the overlap length, and specifically includes:

[0020] Pre-set the segmentation length and overlap length;

[0021] Pre-set the segmentation length and overlap length , according to the formula:

[0022] ;

[0023] Calculate the dynamic offset , sequentially intercept length fragments as segmentation results, adjacent segmentation fragments retain length overlap, obtain the segmentation result set;

[0024] where, represent the sequential number of the current segmentation fragment, represent the preset segmentation length, represent the preset overlap length, represent the dynamic offset of the start position of the segment fragment, represent the dynamic balance factor based on the index and the preset parameter;

[0025] To ensure the continuity and accuracy of segmentation, there is a certain overlap between adjacent segmentation fragments according to the pre-set overlap length.

[0026] Optionally, the S3 further comprises:

[0027] For part of the knowledge text data, manually marked segmentation is used to realize segmentation by manually adding specific identifiers in the text content, including:

[0028] The annotator inserts specific identifiers in the text according to the semantic structure and business requirements of the text, which are used to mark the start and end positions of knowledge blocks and clearly define a complete knowledge unit;

[0029] When loading the text, automatically identify these identifiers and segment the text according to the identifiers to divide it into corresponding large text blocks and small text blocks.

[0030] Optionally, the method further comprises:

[0031] Determine the optimal large language model LLM, submit each large text block to the LLM, and the LLM pre-generate a set of related questions and a summary based on the content of the large text block;

[0032] The optimal embedding model carries out vector embedding processing on the related questions and summary abstracts, and after being converted into vector representations, the related questions and summary abstracts are stored in a vector database together with the large text blocks and corresponding small text blocks, and an index relationship is established therebetween to form the guarantee knowledge library.

[0033] Optionally, the establishing of the index relationship specifically comprises:

[0034] An index table is created for each large text block to record vector IDs of the large text block, the pre-generated related questions and summary abstracts and a corresponding relationship therebetween.

[0035] In another aspect, a guarantee knowledge library construction system based on LLM and RAG technologies is provided, and the system comprises:

[0036] A collection and import module is configured to collect guarantee knowledge data files and import the guarantee knowledge data files into a retrieval enhancement generation (RAG) system to form a preliminary knowledge library.

[0037] A loading and parsing module is configured to load and parse the guarantee knowledge data files in the preliminary knowledge library, store parsed text content in a memory, and form complete knowledge text data.

[0038] A word segmentation module is configured to use an optimized word segmentation strategy to segment the knowledge text data to obtain a plurality of large text blocks and corresponding small text blocks.

[0039] An embedding module is configured to determine and use an optimal embedding model, use the optimal embedding model to carry out vector embedding processing on knowledge blocks composed of the large text blocks and the corresponding small text blocks, store the knowledge blocks in a vector database after being converted into vector representations, and form a guarantee knowledge library.

[0040] In another aspect, an electronic device is provided, and the electronic device comprises a processor and a memory, and the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the guarantee knowledge library construction method based on LLM and RAG technologies.

[0041] In another aspect, a computer-readable storage medium is provided, and the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the guarantee knowledge library construction method based on LLM and RAG technologies.

[0042] The technical solution provided by the present application has at least the following beneficial effects:

[0043] The guarantee knowledge base construction method of the present application not only collects a large amount of guarantee knowledge data, but also uses an optimized word segmentation strategy to obtain multiple large text blocks and corresponding small text blocks, and pre-generates a set of related questions and a summary abstract according to the large text blocks in the data vectorization process, and the related questions and the summary abstract are vector embedded and stored in the vector database together with the large text blocks and the corresponding small text blocks. The knowledge base constructed in this way not only has more accurate data content, but also can capture the query intention of the user from different angles during subsequent user retrieval, increase the probability of recalling relevant documents, and obtain more accurate answers, providing efficient knowledge services and intelligent assistant tools for the guarantee group, such as case analysis, industry research, business assistant, etc., promoting the digital transformation of the guarantee industry, improving business efficiency, risk management ability and customer satisfaction. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 is a guarantee knowledge base construction method flowchart based on LLM and RAG technology provided by an embodiment of the present application;

[0046] Figure 2 is a guarantee knowledge base construction system block diagram based on LLM and RAG technology provided by an embodiment of the present application;

[0047] Figure 3 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0048] Figure 4 is a guarantee knowledge base construction flowchart of the present application;

[0049] Figure 5 is a text block diagram after cleaning of the present application. DETAILED DESCRIPTION

[0050] In order to make the technical problems, technical solutions and advantages of the present application clearer, the following will be described in detail with reference to the drawings and specific embodiments.

[0051] Please refer to Figures 1-5 An embodiment of the present application provides a guarantee knowledge base construction method based on LLM and RAG technology. The method can be realized by an electronic device, which can be a terminal or a server. Figure 1 As shown in the method flowchart, the processing flow can include the following steps:

[0052] S1, collect guarantee knowledge material files and import into a retrieval augmented generation (RAG) system to form a preliminary knowledge base;

[0053] The embodiment of the application collects a large number of bond and bond guarantee, compliance management, financial management and informationization guarantee knowledge material files, and the file material format supports word, pdf, txt, excel, is imported into the retrieval augmented generation (RAG) system after collection, and a preliminary knowledge base is formed.

[0054] The retrieval augmented generation (RAG) system combines information retrieval technology and language generation model, and attempts to solve the problem that the language model lacks real-time accurate knowledge when answering questions. It provides supplementary knowledge for the language model by retrieving relevant information from an external knowledge base, so that the model can more accurately answer questions.

[0055] S2, load and parse the guarantee knowledge material files in the preliminary knowledge base, store the parsed text content in the memory to form complete knowledge text data;

[0056] The embodiment of the application supports loading of knowledge text in multiple common formats such as TXT, DOCX, PDF, etc. During the loading process, the file is verified for legality. If the file format is incorrect, damaged or unreadable, an error message should be recorded and the user should be prompted to upload again. At the same time, in order to improve the loading efficiency, multiple thread technology can be used to process multiple files in parallel.

[0057] The embodiment of the application uses corresponding parsing strategies according to different file formats: for TXT files, the content is read directly by line; for DOCX files, a special document parsing library is used to extract the text content; for PDF files, OCR technology is used to recognize the text and extract it. After parsing, the text content is stored in the memory to form complete knowledge text data.

[0058] S3, using an optimized word segmentation strategy to segment the knowledge text data to obtain multiple large text blocks and corresponding small text blocks;

[0059] Optionally, the S3 specifically includes:

[0060] The knowledge text is segmented using a word segmentation model based on the pre-trained BERT architecture, and according to the logical structure and semantic integrity of the knowledge content, the formula is used:

[0061] ;

[0062] Calculate the block quality score determining a division criterion of the large text block, and obtaining a plurality of large text blocks with reasonable semantic units;

[0063] wherein, represents a semantic coherence aggregation weight, represents a position dispersion compensation coefficient, represents a punctuation density inhibition coefficient, represents a semantic similarity average of the first adjacent sub-block, represents a starting position index of the current candidate sub-block, represents a reference position index of the first semantic anchor point, represents a number of punctuation symbols in the current sub-block, represents a total number of characters in the current sub-block, represents a number of adjacent sub-blocks;

[0064] performing word segmentation on the knowledge text using a word segmentation model based on a pre-trained BERT architecture, specifically, first, preprocessing the knowledge text to be processed, including removing irrelevant characters and standardizing formats, then, taking a sentence terminator (such as a period, a question mark, an exclamation mark) as an initial potential segmentation point, processing a text paragraph containing three sentences “Guarantee refers to the credit guarantee provided by a third party for the debtor to the creditor. Guarantee usually occurs in lending or trading activities, aiming to enhance the security and realization of the creditor's rights. Throughout the process, the guarantor promotes capital flow and maintains market credit order by assessing risks and charging fees.”, the system will evaluate the quality of segmentation between the first and second sentences, at this time, the first candidate text block is “Guarantee refers to the credit guarantee provided by a third party for the debtor to the creditor.”, the number of adjacent sub-blocks is set to 1 (i.e. only considering the semantic association of the previous sub-block), since it is the first sub-block, there is no previous sub-block, the semantic similarity average of the adjacent sub-block is set to the initial value 0.8, through experiments, grid search and cross-validation are performed on a data set containing 5000 technical documents to determine the semantic coherence aggregation weight is 0.58, the position dispersion compensation coefficient is 0.25, the punctuation density inhibition coefficient is 0.12, these numerical combinations achieve the highest ROUGE-L score 0.89 on the test set, the starting position index of the candidate sub-block is 0, the reference position index of the preset semantic anchor point (usually a chapter title or a topic sentence) in the text is also 0, the total number of characters of the sub-block is 28, wherein the number of punctuation symbols is 1, the sub-block quality score is calculated by substituting the formula Subsequently, the system evaluates the quality of merging the first two sentences into a single block. The candidate block at this point is: "Guarantees typically occur in lending or trading activities, aiming to enhance the security and enforceability of claims. Throughout the process, the guarantor facilitates the flow of funds and maintains market credit order by assessing risks and charging fees." The average semantic similarity of its adjacent blocks is [missing value]. The value was obtained by calculating the cosine similarity between the BERT sentence vectors of the first and second sentences, which was 0.92. (Total number of characters in this block) The number of punctuation marks is 65. The value is 2. By iteratively calculating the block quality score of all potential split point combinations, the block is selected such that the average value of all final blocks is 2. The segmentation scheme with the highest value, for example, if we segment only the first sentence... The score is 0.75, while the combined first two sentences... If the score is 0.88, the system tends to favor the latter. In this way, the system determines the criteria for dividing large text blocks based on the logical structure and semantic integrity of the knowledge content, and obtains multiple large text blocks with reasonable semantic units.

[0065] The advantage of the formula lies in introducing a representation of semantic coherence. Items, representing the importance of the structure Items and indicators representing content fluency Items, and combined with adjustable weighting coefficients. , , This enables a multi-dimensional comprehensive evaluation of the quality of text block segmentation.

[0066] Table 1: Example Table of Block Quality Score Calculation

[0067]

[0068] As shown in Table 1, the candidate blocks generated by different segmentation methods are scored. The quality score of the first candidate block, "A guarantee refers to a credit guarantee provided by a third party to a creditor on behalf of a debtor," is as follows. The calculation process is as follows: The second candidate segment, "Guarantees typically occur in lending or trading activities, aiming to enhance the security and enforceability of claims. Throughout the process, the guarantor facilitates the flow of funds and maintains market credit order by assessing risks and charging fees," The calculation process is as follows: In the formula, the summation operation This is used to accumulate the semantic similarity of all adjacent blocks, and then divided by the number of adjacent blocks. To obtain the average similarity, we can reflect the overall semantic coherence of the segmentation, using square root and absolute value operations. The distance between the current block start position and the core semantic anchor point is calculated, the absolute value of the distance ensures that the offset is positive, the square root operation smoothes the influence of distance growth, and the subtraction operation A penalty term is introduced to reduce the weight of the block with too dense punctuation, which may lead to content fragmentation, and the multiplication operation is used to combine the weight coefficients 、 、 The features of different dimensions are fused into the final block quality score The preset block quality score threshold is 0.5, and the score of the second candidate block is 0.53, which is higher than the threshold, indicating that merging the first two sentences into one text block has higher rationality. The numerical result will be used as a decision basis to guide the system to select a better text segmentation point, and finally form a large text block sequence containing complete knowledge fragments.

[0069] After the large text block segmentation is completed, the length of each large text block is judged. If the large text block size exceeds the preset text block length threshold, the length segmentation method is used to perform secondary segmentation on the large text block, and the large text block is subdivided into multiple small text blocks meeting the length requirement, and the association relationship between the small text block and the large text block is established. If the length of the large text block is less than or equal to the preset text block length threshold, the large text block is directly saved as a complete unit in the database.

[0070] The embodiment of the application improves the accuracy of segmentation by optimizing the segmentation strategy, effectively reduces the knowledge block segmentation error, and in the subsequent retrieval recall, through the association relationship between the large text block and the small text block, not only the small text block is recalled, but also the related large text block is recalled, so that the recalled document is more comprehensive and accurate.

[0071] Optionally, the association relationship between the small text block and the large text block is established, and specifically includes:

[0072] The "parent_block_id" field is added in the data structure to record the large text block (parent block) information to which the small text block belongs.

[0073] Optionally, the length segmentation method performs automatic segmentation processing by setting the segmentation length and the overlap length, and specifically includes:

[0074] The segmentation length and the overlap length are preset;

[0075] The segmentation length is preset And the overlap length According to the formula:

[0076] ;

[0077] Computing dynamic offset , sequentially intercepting from the text starting position , length fragments as segmentation results, adjacent segmentation fragments are retained , length overlap part, obtain the segmentation result set;

[0078] , wherein represents the sequential number of the current segmentation fragment, represents the preset segmentation length, represents the preset overlap length, represents the dynamic offset of the starting position of the th segment, represents the dynamic balance factor based on the index and the preset parameter;

[0079] Pre-set segmentation length character and overlap length character, calculate the dynamic offset according to the formula , when , , intercept the fragment from the starting position 0 to 8 characters, when , , intercept the fragment from the starting position 5.25 to 13.25 characters (rounded to 5 to 13 characters), there are 3 characters overlapping in the 5-8 character interval between adjacent fragments, when , , intercept the fragment from 10.5 to 18.5 characters (rounded to 10 to 18 characters), and the previous fragment overlaps 3 characters in the 10-13 character interval, through the dynamic offset calculation, ensure that the adjacent fragments retain the preset overlap length, and through the root value and absolute value operation, compensate the index offset error, finally obtain the continuous and complete text segmentation result set.

[0080] Table 2: Dynamic offset calculation example table

[0081]

[0082] As shown in Table 2, the dynamic offset is obtained by formula calculation, wherein the root value term is when , and is when , the compensation term value is dynamically adjusted with the index increasing, when , the calculation result contains a decimal, the intercept interval adopts the down rounding processing, to ensure that the character index is an integer, and the overlap length of adjacent fragments is always maintained at 3 characters, through the dynamic compensation mechanism, the actual overlap area fluctuates between 3-4 characters, improving the fault tolerance of the segmentation boundary.

[0083] The parameters in the formula According to the experimental determination of the average semantic unit length of the text, the statistics of 100 news texts show that 8 characters cover 80% of the independent semantic units, and the parameter is set to 37.5% (i.e. 3 characters). Through cross-validation testing, this proportion balances the recall rate and the accuracy rate. The square root term is used to compensate for the cumulative error of linear deviation caused by index growth. The absolute value operation ensures that the compensation amount is non-negative. The denominator normalizes the compensation amount to prevent overcompensation. For example, when the compensation term is , and the dynamic deviation increment stabilizes at 5.25 / segment. This formula realizes nonlinear deviation compensation through the combination of multiplication, subtraction, square root, and absolute value operations. The final segmentation result set covers the complete text and the error in the overlapping area is controlled within ±1 character.

[0084] To ensure the continuity and accuracy of segmentation, there will be a certain overlap between adjacent segmented segments according to the pre-set overlap length (for example, if the segmentation length is set to 20 words and the overlap length is set to 5 words, the first 1-20 words will be extracted as the first segmented segment, and then 20 words (i.e. 16-35 words) will be extracted from the 16th word as the second segmented segment, and so on, until the segmentation of the entire text is completed).

[0085] Optionally, the S3 further comprises:

[0086] For part of the knowledge text data, manual marking segmentation is used to realize segmentation by manually adding specific identifiers in the text content, including:

[0087] The annotators insert specific identifiers in the text according to the semantic structure and business requirements of the text, which are used to mark the start and end positions of knowledge blocks and clearly define a complete knowledge unit (for example, in a legal text document, the annotators can mark different legal provisions as large text blocks and mark specific paragraphs in the provisions as small text blocks by adding different identifiers (such as <kcs>: large text block, <tcs>The small text block) to achieve accurate segmentation);

[0088] When loading the text, these marks are automatically identified, and the text is segmented according to the marks, divided into corresponding large text blocks and small text blocks.

[0089] The biggest advantage of manual marking segmentation is that the segmentation accuracy is extremely high. Since it is manually marked according to semantics and business logic, it can best meet the actual needs and ensure that the segmentation result meets the expectations. However, the workload is huge, so the embodiment of the application adopts a multi-segmentation scheme parallel strategy to preferentially use a segmentation model for large and small text segmentation. For cases where the recall effect of part of the knowledge text is not ideal, manual marking segmentation can be considered for appropriate use.

[0090] S4, determine and use the optimal embedding model, and perform vector embedding processing on the knowledge block composed of the large text block and the corresponding small text block, store it in the vector database after conversion to vector representation, and form a guarantee knowledge library.

[0091] According to the guarantee knowledge material file, the embodiment of the application determines bge-large-zh-v1.5 as the optimal embedding model through replacement testing, and performs vector embedding processing on the knowledge block composed of the large text block and the corresponding small text block, and stores it in the vector database after conversion to vector representation, and forms a guarantee knowledge library.

[0092] Optionally, the method further comprises:

[0093] Determine the optimal large language model LLM (the embodiment of the application determines the large chat robot model glm4-9b-chat developed by Zhisu AI as the optimal large language model LLM through replacement testing), submit each large text block to the LLM, and the LLM pre-generates a set of related questions and a summary based on the content of the large text block (for example, for a knowledge block related to the guarantee business process, generate questions such as "What are the application conditions for the guarantee business?", "What are the key links of the guarantee approval?" and a brief summary of the business process);

[0094] The optimal embedding model performs vector embedding processing on the related questions and summary, converts it to vector representation, and stores it in the vector database together with the large text block and the corresponding small text block, and establishes an index relationship between them, forming the guarantee knowledge library.

[0095] The embodiment of the present application can not only generate corresponding vectors according to the query statement input by the user when the user asks follow-up questions, but also match from multiple dimensions, in addition to matching with the vectors of the original knowledge block data, the vectors of the pre-generated related questions and the summary vectors are also matched, through this multi-dimensional matching method, even if there is a certain information loss in the data vectorization process, the query intention of the user can be captured from different angles, the probability of retrieving relevant documents is increased, thereby effectively reducing the problem of inaccurate retrieval caused by information loss.

[0096] Optionally, the establishing of the index relationship therebetween specifically comprises:

[0097] An index table is created for each large text block to record the vector IDs of the large text block, the pre-generated related questions and the summary and the corresponding relationship therebetween.

[0098] As shown in Figure 2 The embodiment of the present application also provides a guarantee knowledge library construction system based on LLM and RAG technology, the system comprises:

[0099] A collection and import module 210 is used to collect guarantee knowledge data files and import them into a retrieval enhancement generation RAG system to form a preliminary version of a knowledge base;

[0100] A loading and parsing module 220 is used to load and parse the guarantee knowledge data files in the preliminary version of the knowledge base, store the parsed text content in the memory to form complete knowledge text data;

[0101] A word segmentation module 230 is used to segment the knowledge text data using an optimized word segmentation strategy to obtain a plurality of large text blocks and corresponding small text blocks;

[0102] An embedding module 240 is used to determine and use an optimal embedding model, and the knowledge blocks composed of the large text blocks and the corresponding small text blocks are subjected to vector embedding processing using the optimal embedding model, and are stored in a vector database after being converted into vector representations to form a guarantee knowledge library.

[0103] The guarantee knowledge library construction system based on LLM and RAG technology provided by the embodiment of the present application corresponds to the guarantee knowledge library construction method based on LLM and RAG technology provided by the embodiment of the present application in terms of functional structure, and will not be described here.

[0104] Figure 3 is a structural schematic diagram of an electronic device 300 provided by an embodiment of the present application. The electronic device 300 can have great differences due to different configurations or performances, and can include one or more processors (central processing units, CPUs) 301 and one or more memories 302. The memory 302 stores at least one instruction, which is loaded and executed by the processor 301 to implement the steps of the above-mentioned method for constructing a guarantee knowledge base based on LLM and RAG technology.

[0105] In an exemplary embodiment, a computer-readable storage medium, such as a memory including instructions executable by a processor in a terminal to complete the above-mentioned method for constructing a guarantee knowledge base based on LLM and RAG technology, is also provided. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0106] Figure 4 is a guarantee knowledge base construction process.

[0107] Data collection: Collect data related to guarantee business through multiple channels, including various text form data such as guarantee business case documents and industry research reports, as basic data sources for constructing a guarantee knowledge base.

[0108] Text cleaning: Perform text cleaning operations on the collected data to remove noise data (such as irrelevant characters, incorrect encoding, repeated content, etc.) in the text, standardize the text format, and improve the data quality to provide clean and effective text content for subsequent processing. The cleaned text is stored in the guarantee knowledge base.

[0109] Text segmentation: Segment the text in the guarantee knowledge base according to the rules of semantic association and content theme, and divide the long text into multiple guarantee knowledge blocks to make the knowledge more granular and facilitate subsequent targeted processing and knowledge mining.

[0110] Text vectorization: Perform text vectorization conversion on the guarantee knowledge blocks obtained by segmentation, and use word embedding and vector encoding technologies in natural language processing to convert the text into a vector form that can be recognized and easily calculated and retrieved by a computer. The generated vector data is stored in the guarantee knowledge vector library, realizing the structured and numerical representation of knowledge, and facilitating the rapid matching, retrieval and intelligent application of knowledge.

[0111] Figure 5 is a schematic diagram of text segmentation processing results, showing the structured organization form of the cleaned text:

[0112] Two types of text blocks are distinguished: KCS (large text blocks): as an upper container, multiple TCSs can be nested, and multi-level KCS nesting structures are shown in the figure, reflecting the macro grouping of text. TCS (small text block): as a basic unit, it carries specific text content (such as numbered items 1-6), and is the basic carrier of text semantics.

[0113] Through hierarchical nesting (KCS contains TCS, KCS can be nested in KCS), the organization architecture of the cleaned text from the whole (large module) to the local (small unit) is clearly displayed, reflecting the structured splitting and classification logic of the text block algorithm on the content, which is commonly used in NLP processes such as text mining and information extraction, to assist subsequent semantic analysis and data processing.

[0114] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium, which can be a read-only memory, a magnetic disk or an optical disk.

[0115] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.< / tcs> < / kcs>

Claims

1. A method for constructing a guaranteed knowledge library based on LLM and RAG technologies, characterized in that, The method comprises: S1, collect the guarantee knowledge data files and import them into the retrieval enhancement generation RAG system to form a preliminary knowledge base; S2, load and parse the guarantee knowledge data files in the preliminary knowledge base, store the parsed text content in the memory to form complete knowledge text data; S3, use an optimized word segmentation strategy to segment the knowledge text data to obtain a plurality of large text blocks and corresponding small text blocks; S4, determine and use the optimal embedding model to perform vector embedding processing on the knowledge blocks composed of large text blocks and corresponding small text blocks, store them in the vector database after conversion into vector representation to form a guarantee knowledge library; The S3 specifically comprises: using a word segmentation model based on the pre-trained BERT architecture to segment the knowledge text, and using the formula according to the logical structure and semantic integrity of the knowledge content: ； Computing chunk quality scores determining a division criterion of the large text block to obtain a plurality of large text blocks with reasonable semantic units; in, Represents the aggregation weight of semantic coherence. Represents the positional discrepancy compensation coefficient. Represents the punctuation density suppression coefficient. Representing the The average semantic similarity of adjacent blocks Represents the starting position index of the current candidate block. Representing the The baseline position index of each semantic anchor point This represents the number of punctuation marks within the current block. Represents the total number of characters in the current block. Represents the number of adjacent blocks; After the large text block is segmented, the length of each large text block is judged, if the size of the large text block exceeds the pre-set text block length threshold, a length segmentation method is used to further segment the large text block into a plurality of small text blocks meeting the length requirement, and an association between the small text blocks and the large text block is established, if the length of the large text block is less than or equal to the pre-set text block length threshold, the large text block is directly saved as a complete unit.

2. The method of claim 1, wherein, The association between the small text blocks and the large text block is specifically established by: adding a "parent_block_id" field in the data structure to record the large text block information to which the small text block belongs.

3. The method of claim 1, wherein, The length segmentation method performs automatic segmentation processing by setting the segmentation length and the overlap length, specifically comprising: pre-setting the segmentation length and the overlap length; Pre-set word segmentation length and overlap length , according to the formula: ； Computing dynamic offsets Taking the text starting position in turn Length fragments as a result of word segmentation, adjacent word segmentation fragments are retained Length coincides with the obtained word segmentation result set; wherein, a sequence number representing a current segment, a preset segment length, a preset overlap length, a dynamic offset representing a start position of the first segment, a dynamic offset representing a start position of the first segment, a dynamic balance factor based on the index and the preset parameter; To ensure the coherence and accuracy of segmentation, there is a certain overlap between adjacent segmented fragments according to the pre-set overlap length.

4. The method of claim 1, wherein, The S3 further comprises: For part of the knowledge text data, manual marking segmentation is used to realize segmentation by manually adding specific marks in the text content, including: annotators insert specific marks in the text according to the semantic structure and business requirements of the text to mark the start and end positions of the knowledge blocks and clearly define a complete knowledge unit; When loading the text, these marks are automatically recognized, and the text is segmented into corresponding large text blocks and small text blocks according to the marks.

5. The method of claim 1, wherein, The method further comprises: determining the optimal large language model LLM, submitting each large text block to the LLM, and the LLM pre-generates a set of related questions and a summary based on the content of the large text block; The optimal embedding model performs vector embedding processing on the related questions and the summary, converts them into vector representation, and stores them in the vector database together with the large text blocks and the corresponding small text blocks, and establishes an index relationship between them to form the guarantee knowledge library.

6. The method of claim 5, wherein, The index relationship between them is specifically established by: creating an index table for each large text block to record the vector IDs of the large text blocks, the pre-generated related questions and the summary, and the corresponding relationship between them.

7. A system for building a secured knowledge repository based on LLM and RAG technologies, said system being configured to implement the method of any one of claims 1-6, characterized in that, The system comprises: The collection and import module is configured to collect the guarantee knowledge data files and import them into the retrieval augmentation generation (RAG) system to form a preliminary knowledge base. The loading and parsing module is configured to load and parse the guarantee knowledge data files in the preliminary knowledge base, store the parsed text content in the memory, and form complete knowledge text data. The word segmentation module is configured to use an optimized word segmentation strategy to segment the knowledge text data to obtain a plurality of large text blocks and corresponding small text blocks. The embedding module is configured to determine and use an optimal embedding model, use the optimal embedding model to perform vector embedding processing on the knowledge blocks composed of the large text blocks and the corresponding small text blocks, store the converted vector representations in a vector database, and form a guarantee knowledge library. 8.An electronic device comprising a processor and a memory in which at least one instruction is stored, wherein, The at least one instruction is loaded and executed by the processor to implement the guarantee knowledge library construction method based on the LLM and RAG technologies according to any one of claims 1-6.

9. A computer-readable storage medium having stored therein at least one instruction, wherein The at least one instruction is loaded and executed by the processor to implement the guarantee knowledge library construction method based on the LLM and RAG technologies according to any one of claims 1-6.

Images