Text comparison methods, devices, equipment, and media based on large language models

Abstract

This application provides a text comparison method, apparatus, device, and medium based on a large language model. The method includes: using a large language model to acquire multiple first keypoint information from a first text to be compared, and determining the location range of the first keypoint information; based on the first keypoint information, acquiring non-repeating keypoint information corresponding to each location range, and calculating its first quantization information; based on a comparison vector database corresponding to a second text to be compared, retrieving second quantization information matching the first quantization information, and recording comparison information where the logical text block corresponding to the second quantization information is a non-repeating keypoint; based on the non-repeating keypoint information and the comparison information, using the large language model to acquire semantic comparison results; and based on the semantic comparison results, outputting the text comparison results of the first and second texts to be compared. This application uses a large language model and improves the accuracy and comprehensiveness of text comparison through semantic analysis.

Description

Technical Field

[0001] This application relates to the field of text processing technology, and in particular to a text comparison method, apparatus, device and medium based on a large language model. Background Technology

[0002] With the development of information technology, enterprises and organizations are facing an ever-increasing volume of documents, often involving complex content management across multiple departments and versions. Document comparison technology, by matching and analyzing the content of documents from different versions or sources, helps users quickly identify differences, omissions, and conflicts between texts, thereby ensuring content consistency and integrity. For example, document comparison technology can help ensure that internal documents accurately implement the requirements of external regulatory policies. By comparing regulatory policies with internal guidance documents, potential omissions can be identified in a timely manner, reducing economic losses caused by inadequate policy implementation. Furthermore, document comparison technology can help users quickly identify changes between old and new documents, ensuring the update and accuracy of the knowledge base.

[0003] In existing technologies, document comparison methods can be mainly divided into character-based, word-based, and simple semantic-based comparison methods. Traditional methods are based on character or word matching techniques, such as edit distance and word frequency statistics. These methods measure similarity by calculating the number of insertions, deletions, and replacements of characters or words between texts. This method is simple to implement, computationally efficient, and performs well when processing plain text or structured data. However, this method only focuses on changes at the character and word level and lacks an understanding of semantic relationships. When two texts differ significantly in literal expression but are semantically similar, this method cannot effectively identify their similarity.

[0004] To overcome the limitations of character matching, some existing technologies have introduced comparison methods based on keyword and phrase matching. These methods extract key terms or phrases from the text and perform centralized comparison and statistics. This approach can improve the semantic relevance of the comparison to some extent. However, these methods rely on predefined sets of keywords or rules, making it difficult to adapt to multi-domain or diverse text content.

[0005] Therefore, there is an urgent need for a text comparison method to solve the technical problem that existing technologies are prone to misjudgment or omission during text comparison. Summary of the Invention

[0006] This application provides a text comparison method, apparatus, device, and medium based on a large language model to solve the technical problem of misjudgment or omission during text comparison, and to achieve more accurate and comprehensive text comparison.

[0007] Firstly, this application provides a text comparison method based on a large language model, including:

[0008] Multiple first logical text blocks corresponding to the first text to be compared are obtained, and according to the preset prompt words, a large language model is used to obtain the first key point information corresponding to each first logical text block, and the position range of the first key point information in the first text to be compared is determined.

[0009] Based on multiple first key point information within the same location range, according to the preset key point information processing rules, obtain non-repeating key point information corresponding to each location range, and calculate the first quantitative information of each non-repeating key point information.

[0010] Obtain multiple second logical text blocks corresponding to the second text to be compared, and based on the comparison vector database generated from the second logical text blocks, retrieve second quantization information that matches the first quantization information, and record the comparison information where the second logical text block corresponding to the second quantization information is the non-repeating key point;

[0011] Based on the non-repeating key point information and the comparison information of the non-repeating key points, a large language model is used to obtain the semantic comparison results of the non-repeating key point information and the comparison information.

[0012] Iterate through all the non-repeating key point information and, based on all the semantic comparison results, output the text comparison results of the first text to be compared and the second text to be compared, thus completing the text comparison.

[0013] Optionally, in the method described above, the step of obtaining non-repeating key point information corresponding to each location range based on multiple first key point information within the same location range, according to a preset key point information processing rule, and calculating the first quantization information of each non-repeating key point information, includes:

[0014] Filter and obtain multiple first key point information belonging to the same location range, use a preset semantic similarity model to identify and merge similar first key point information to obtain the second key point information corresponding to each location range;

[0015] The semantic similarity model is a logistic regression model based on edit distance and vector cosine similarity.

[0016] Based on the location range, the second key point information is spliced ​​together to obtain non-repeating key point information corresponding to each location range;

[0017] Based on the non-repeating key point information, an embedding technique is used to quantize and obtain the first quantization information of each non-repeating key point information.

[0018] Optionally, in the method described above, the step of obtaining the semantic comparison result of the non-repeating keypoint information and the comparison information to be compared using a large language model based on the non-repeating keypoint information and the comparison information to be compared includes:

[0019] Based on the non-repeating key point information and the information to be compared in each group, a large language model with preset prompt words is used to calculate and obtain the semantic similarity between the non-repeating key point information and the information to be compared.

[0020] If the semantic similarity between the non-repeating key point information and the information to be compared is greater than or equal to a similarity threshold, then it is determined that the information to be compared and the non-repeating key point information express the same meaning; or,

[0021] If the semantic similarity between the non-repeating key point information and the information to be compared is less than a similarity threshold, then it is determined that the information to be compared and the non-repeating key point information express different meanings.

[0022] Optionally, in the method described above, the step of traversing all the non-repeating key point information and outputting the text comparison results of the first text to be compared and the second text to be compared based on all the semantic comparison results includes:

[0023] Traverse all the non-repeating key point information and obtain the semantic comparison results corresponding to all the non-repeating key point information.

[0024] After determining that the information to be compared and the non-repeating key point information express the same meaning, the non-repeating key point information is marked as a covered item and stored in the covered dataset;

[0025] After determining that the information to be compared and the non-repeating key point information express different meanings, the non-repeating key point information is marked as an uncovered item and stored in the uncovered dataset;

[0026] Based on the covered dataset and / or uncovered dataset, output the text comparison results of the first text to be compared and the second text to be compared, and complete the text comparison.

[0027] Optionally, the construction of the comparison vector database, as described above, includes:

[0028] Obtain multiple second logical text blocks of the second text to be compared, and generate second quantization information corresponding to each second logical text block using a preset embedding technique based on all the second logical text blocks;

[0029] All the second quantization information is stored in the comparison vector database, and an index is created on the second quantization information;

[0030] Based on the comparison vector database generated from the second logical text block, second quantization information matching the first quantization information is retrieved, and comparison information recording the second logical text block corresponding to the second quantization information as the non-repeating key point is recorded, including:

[0031] Based on the first quantization information, a search is performed in the comparison vector database to obtain a second quantization information set that matches the first quantization information.

[0032] Based on the second quantization information in the second quantization information set, obtain the second logical text block corresponding to each second quantization information, and record the second logical text block as the comparison information of the non-repeating key points.

[0033] Optionally, in the method described above, the step of obtaining the first key point information corresponding to each first logical text block using a large language model based on preset prompt words, and determining the position range of the first key point information in the first text to be compared, includes:

[0034] Based on the preset prompt words, according to the first logical text block, the large language model is used to obtain the first key point information associated with the preset prompt words;

[0035] Traverse the first set of logical text blocks and obtain the first key point information corresponding to each first logical text block;

[0036] Based on the preset text structure of the first text to be compared, the location information of the first key point information is obtained, and based on the location information, the location range of the first key point information in the first text to be compared is determined.

[0037] Optionally, the method described above further includes:

[0038] The file to be compared is parsed and preprocessed to obtain the text information of the file to be compared.

[0039] Based on the text information, a trained block model is used to obtain multiple logical text blocks of the text information;

[0040] The block model includes at least one of the BERT model and the SemanticChunker model;

[0041] The step of obtaining multiple first logical text blocks corresponding to the first text to be compared includes:

[0042] The first file to be compared is parsed and preprocessed to obtain the text information of the first text to be compared.

[0043] Based on the text information of the first text to be compared, a trained BERT model is used to obtain multiple first logical text blocks of the first text to be compared.

[0044] Then, obtain multiple second logical text blocks corresponding to the second text to be compared, including:

[0045] The second file to be compared is parsed and preprocessed to obtain the text information of the second file to be compared.

[0046] Based on the text information of the second file to be compared, a trained BERT model is used to obtain multiple second logical text blocks of the second file to be compared.

[0047] Secondly, embodiments of this application provide a text comparison device, comprising:

[0048] The key point information acquisition module is used to acquire multiple first logical text blocks corresponding to the first text to be compared, and according to the preset prompt words, adopts a large language model to acquire the first key point information corresponding to each first logical text block, and determines the position range of the first key point information in the first text to be compared.

[0049] The information quantification module is used to obtain non-repeating key point information corresponding to each location range based on multiple first key point information within the same location range, according to preset key point information processing rules, and to calculate the first quantification information of each non-repeating key point information.

[0050] The information retrieval module is used to obtain multiple second logical text blocks corresponding to the second text to be compared, and based on the comparison vector database generated from the second logical text blocks, retrieve second quantization information that matches the first quantization information, and record the comparison information that the second logical text block corresponding to the second quantization information is the non-repeating key point.

[0051] The information comparison module is used to obtain the semantic comparison results of the non-repeating key point information and the comparison information to be compared using a large language model, based on the non-repeating key point information and the comparison information to be compared.

[0052] The result output module is used to traverse all the non-repeating key point information and, based on all the semantic comparison results, output the text comparison results of the first text to be compared and the second text to be compared, thus completing the text comparison.

[0053] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0054] The memory stores computer-executed instructions;

[0055] The processor executes computer execution instructions stored in the memory to implement the above-described method.

[0056] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the above-described method.

[0057] The text comparison method, apparatus, device, and medium based on a large language model provided in this application achieve efficient comparison of complex texts through semantic analysis and comparison, improving the accuracy and comprehensiveness of text comparison. First, by segmenting the first text to be compared into blocks, and combining preset prompt words and a large language model, the first key point information of each logical text block can be efficiently extracted, and its position range in the original text can be located. This process ensures the comprehensiveness and accuracy of the comparison benchmark, laying the foundation for the comparison task. Simultaneously, through semantic segmentation and prompt word guidance, interference from redundant information can be effectively reduced, improving the quality of key information extraction. Second, for multiple first key point information within the same position range, preset key point information processing rules are used to obtain non-repeating key point information and perform quantification processing. This method not only achieves semantic deduplication of redundant information but also generates first quantified information representing key point information. This quantification transforms the semantic features of the text, facilitating comparison. This step improves the refinement of the comparison data, ensuring that each piece of information is unique and representative in the comparison.

[0058] During the comparison phase, by segmenting the second text to be compared into blocks and generating a comparison vector database, the second quantitative information matching the first quantitative information can be retrieved efficiently. This process, based on vectorized semantic matching technology, shortens the comparison time and improves the accuracy of the matching results. The extraction of the comparison information provides support for further semantic comparison, enabling the comparison to focus on the core content between the texts. Next, using a large language model, semantic comparison is performed on non-repeating key information and its corresponding comparison information to determine the semantic consistency between the two texts. This semantic comparison overcomes the limitations of traditional character matching, not only identifying surface similarities but also capturing deeper semantic relationships, thus ensuring the comprehensiveness and accuracy of the comparison results. Finally, by traversing all non-repeating key information, a final text comparison report is generated based on the comparison results. The report can intuitively reflect the differences in content coverage between the first and second texts to be compared. It can quickly identify uncovered key information, thereby discovering possible omissions or inconsistencies in the text. This comparison method improves the efficiency and reliability of text comparison. In summary, by combining large language models and quantization matching technology, this application achieves accurate analysis and comparison of semantic relationships between texts, which not only improves the accuracy and comprehensiveness of the comparison, but also shortens the comparison time and improves the efficiency and quality of processing large-scale text tasks. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0060] Figure 1 A flowchart illustrating a text comparison method based on a large language model provided in this application embodiment;

[0061] Figure 2 This is a schematic flowchart of a method for obtaining first quantization information provided in an embodiment of this application;

[0062] Figure 3 A schematic flowchart illustrating the method for obtaining semantic comparison results of non-duplicative key point information and comparison information provided in an embodiment of this application;

[0063] Figure 4 This is a schematic diagram of the method for outputting text comparison results provided in an embodiment of this application;

[0064] Figure 5 This is a schematic diagram of the method for constructing a comparison vector database provided in an embodiment of this application;

[0065] Figure 6 A schematic flowchart illustrating a method for retrieving and filtering second quantitative information and obtaining comparison information provided in an embodiment of this application;

[0066] Figure 7 This is a flowchart illustrating a method for determining the location range of the first key point information provided in an embodiment of this application.

[0067] Figure 8 This is a schematic flowchart of a method for obtaining logical text blocks provided in an embodiment of this application;

[0068] Figure 9 A schematic diagram illustrating the overall process of comparing internal guidance documents and regulatory documents provided for embodiments of this application;

[0069] Figure 10 A schematic diagram illustrating the process of extracting and merging key information from regulatory documents, provided for embodiments of this application;

[0070] Figure 11 A flowchart illustrating the process of retrieving and outputting comparison results of regulatory documents and internal guidance documents provided in the embodiments of this application;

[0071] Figure 12 This is a schematic diagram of the structure of the text comparison device provided in the embodiments of this application;

[0072] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0073] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0074] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0075] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0076] In the management process of enterprises and institutions, relevant regulatory departments usually formulate and issue regulatory policies or guidelines to strengthen control over safety and development direction. Enterprise or institutional management departments typically need to deeply interpret and break down these regulatory policies and guidelines based on their own development status and business divisions, forming multiple internal guidance documents, which are then disseminated downwards for implementation. Currently, the comparison of regulatory documents with internal guidance documents largely relies on manual work. Specifically, staff need to check item by item whether the internal guidance documents fully cover all requirements and clauses in the regulatory documents. This manual comparison method is often time-consuming and prone to omissions or misjudgments when dealing with lengthy and complex regulatory documents. Existing text comparison tools typically use string matching algorithms based on edit distance or calculate cosine similarity by extracting keywords to determine text similarity. These methods focus more on literal similarity and lack an understanding of the deep semantics of the text and the extraction of key information points, making it difficult to meet the needs of comparing regulatory documents with internal guidance documents.

[0077] Based on the aforementioned technical problems and needs, the inventive concept of this application lies in utilizing large language models and quantized matching to achieve efficient and accurate text comparison, overcoming the shortcomings of existing technologies in semantic understanding and comparison efficiency. First, by parsing the first and second texts to be compared into multiple logical text blocks, each text block can be semantically independent, ensuring the accuracy and completeness of text block segmentation and laying the foundation for subsequent key point extraction. Second, within each logical text block, the first key point information is extracted using a large language model, and its specific range in the original text is determined by combining it with location information. To reduce repetitive and redundant information, a semantic similarity model is introduced, aggregating similar key points through preset rules to generate non-repeating key point information. This information is quantized into first quantized information through embedding technology, providing an efficient and accurate numerical basis for comparison.

[0078] Next, for the second text to be compared, a comparison vector database is constructed, the second logical text block is converted into second quantized information, and an index structure is established. By matching the first and second quantized information, comparison information associated with non-repeating key point information is quickly filtered out. This process combines vectorization representation and retrieval techniques to achieve rapid matching in large-scale text comparison. Finally, a large language model is used to perform semantic comparison between the non-repeating key point information and the comparison information, calculate their semantic similarity, and generate a detailed text comparison report based on the comparison results. This report can intuitively reflect the differences in content coverage between the two texts, providing users with clear review results and content consistency verification. It is evident that this application, by integrating a large language model, vectorization techniques, and efficient retrieval strategies, achieves comparison of complex texts, not only improving comparison efficiency but also enhancing the accuracy of semantic comparison, meeting the practical needs of efficiently processing large-scale texts.

[0079] This application is applicable to fields requiring efficient processing and comparison of large amounts of text content. Firstly, in compliance reviews, enterprises and institutions need to ensure consistency between their internal documents and external regulatory policies or industry standards. This application can quickly identify whether internal documents have omissions or deviations from external regulatory requirements, providing timely and accurate content review support and reducing the frequency of issues. Secondly, this application is also applicable to contract review and management scenarios. In the daily operations of large enterprises, comparing contract terms and reviewing versions is a high-frequency and critical task. This application can efficiently compare the content differences between different versions of contracts, quickly identify omissions or modifications, and thus improve review efficiency. This application can also play a role in knowledge base management and updates. For example, when updating technical documents or product manuals, this application can quickly compare old and new versions, identify updated parts, and ensure timely maintenance and version consistency of the knowledge base. This can improve the work efficiency of R&D teams, technical support, and customer service departments. Finally, this application can also be applied to academic research and information retrieval fields, such as paper plagiarism checking and literature review writing. The semantic comparison function in this application can help researchers quickly screen and compare literature content, ensuring the comprehensiveness and accuracy of citations.

[0080] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0081] Figure 1 This is a flowchart illustrating a text comparison method based on a large language model, provided as an embodiment of this application. Figure 1 As shown, the method includes:

[0082] S11, obtain multiple first logical text blocks corresponding to the first text to be compared, and according to the preset prompt words, use a large language model to obtain the first key point information corresponding to each first logical text block, and determine the position range of the first key point information in the first text to be compared.

[0083] In this embodiment, to facilitate subsequent processing, the first text to be compared needs to be divided into several logically coherent text blocks. For example, the division of text blocks can be based on semantic information, chapter structure, or other logical relationships to ensure that each first logical text block has a complete semantic unit. For instance, when the first text to be compared is a policy, regulation, or guidance document issued by a regulatory agency, its content often covers specific requirements in multiple areas. Such documents typically have complex structures, including chapters, clauses, sub-clauses, and supplementary explanations. Therefore, to achieve efficient comparison, the original text needs to be divided and decomposed into multiple logical text blocks with independent semantics.

[0084] After acquiring the first logical text block, pre-defined prompts are used to process each text block using a large language model. The prompts are designed to guide the large language model to focus on the core information in the comparison task. Prompts can be general or customized based on specific application scenarios. For example, in a policy document comparison scenario, prompts such as "Extract the main clauses from the following text" or "List the key content of the following paragraphs" can be used. Large language models (such as Llama, Chat GLM, or other fine-tuned models) can extract representative first key information from complex text. Through their contextual understanding capabilities, large language models can effectively process long texts and accurately identify key information points, improving the efficiency and accuracy of information processing. Examples include the core content of clauses, key indicators of policy requirements, or specific implementation details.

[0085] After obtaining the first key information, its location within the first text to be compared is determined. This step aims to maintain the correlation between the key information and the original text during subsequent comparisons, facilitating traceability and verification. Through these steps, the parsing and key information extraction of the first text to be compared are achieved, laying a solid foundation for subsequent text comparisons. This process fully utilizes the natural language processing capabilities of the large language model, improving the accuracy of key information extraction and ensuring the reliability of the comparison results. This method ensures high accuracy and flexibility throughout the entire process from the original text to logical text blocks and then to key information extraction. It not only efficiently extracts key information from the text but also provides necessary data support and location information for subsequent comparison steps, enhancing the reliability of the comparison method.

[0086] S12, based on multiple first key point information within the same location range, according to the preset key point information processing rules, obtain non-repeating key point information corresponding to each location range, and calculate the first quantization information of each non-repeating key point information.

[0087] In this embodiment, to improve the accuracy and efficiency of the comparison, further processing is required for multiple first key point information extracted within the same location range. First, multiple first key point information belonging to the same location range are aggregated. These key point information may differ in expression, but they have a high degree of semantic similarity or overlap. Therefore, a preset key point information processing rule is used to filter out non-repeating key point information from multiple first key point information within the same location range. The key point information processing rule can be based on various technical means to ensure accurate identification and merging of semantically similar content. This rule not only needs to consider character matching at the text surface level but also needs to deeply analyze the semantic connotation of the text to eliminate duplicate or redundant information. Through this processing, it is ensured that the key point information within each location range is semantically unique and representative, thereby effectively reducing information redundancy.

[0088] After acquiring unique key information, further quantization processing is performed. This quantization process maps the text content to a high-dimensional vector space, transforming each unique key information into a numerical semantic representation, i.e., the first quantization information. This quantization not only accurately expresses the semantic features of the text but also provides an efficient and precise numerical foundation for subsequent comparison operations. Through this processing, this application can effectively extract and optimize key information in complex text environments, providing solid support for efficient matching and retrieval during the comparison process. This method improves the accuracy of text comparison and also lays the foundation for information management in large-scale text processing scenarios.

[0089] S13, obtain multiple second logical text blocks corresponding to the second text to be compared, and based on the comparison vector database generated from the second logical text blocks, retrieve the second quantization information that matches the first quantization information, and record the comparison information where the second logical text block corresponding to the second quantization information is a non-repeating key point.

[0090] In this embodiment, to achieve analysis and accurate comparison of the second text to be compared, it is first necessary to parse it to obtain multiple corresponding second logical text blocks. In the previous example, assuming the first text to be compared was a regulatory document, the second text to be compared is typically a guidance document formed by an enterprise or organization after interpreting and refining the regulatory document. These documents may differ from the first text to be compared in content, but their goal is to maintain consistency with the regulatory document. Therefore, dividing the second text to be compared into multiple logical text blocks with independent semantics can effectively improve the accuracy of subsequent comparison operations. The acquisition of second logical text blocks can be achieved in various ways. Initial division can be made based on the document's structural features, such as chapters, clauses, and paragraphs. To improve the accuracy of the division, pre-trained models or semantic analysis tools can also be used to parse the text, ensuring that each text block is semantically independent and complete.

[0091] After obtaining the second logical text block, a comparison vector database needs to be constructed. Each second logical text block is converted into a corresponding numerical representation, i.e., second quantized information. This quantized information is stored in the form of a high-dimensional vector, which can effectively capture the semantic features of the text. All second quantized information is stored in the comparison vector database. Next, based on the first quantized information, a search is performed in the comparison vector database to find second quantized information that is semantically similar. During the search process, the similarity between the first quantized information and each piece of second quantized information in the database is calculated. Similarity calculation can use various algorithms, such as cosine similarity and Euclidean distance, to measure the closeness of two vectors in the semantic space. When the similarity reaches a preset threshold, the second quantized information is considered to match the first quantized information.

[0092] For each retrieved second quantification information, its corresponding second logical text block is recorded as the comparison information to be associated with the non-repeating key point information. This step associates the non-repeating key point information in the first comparison text with the relevant content in the second comparison text, laying the foundation for subsequent comparisons. This embodiment, by constructing a comparison vector database and performing retrieval, achieves rapid location of content matching the non-repeating key point information in large-scale text data, improving comparison efficiency. Simultaneously, it ensures the accuracy of the comparison, providing strong support for subsequent semantic comparison using a large language model and the final text comparison result output, ensuring the comprehensiveness and reliability of the comparison results.

[0093] S14. Based on the non-repeating key point information and the non-repeating key point information to be compared, a large language model is used to obtain the semantic comparison results of the non-repeating key point information and the information to be compared.

[0094] In this embodiment, to more accurately determine the content coverage between two texts to be compared, semantic comparison of non-repeating key information and its corresponding comparison information is required. To this end, a large language model is used to parse both and obtain the semantic comparison results. First, for each non-repeating key information and its corresponding comparison information, a text pair is formed and input into the large language model for processing. The large language model utilizes its natural language understanding capabilities to perform multi-dimensional semantic analysis on the input text pairs. By deeply analyzing the semantic meaning, syntactic structure, and contextual relationships of words, the large language model can effectively identify potential semantic relationships between texts. During this process, preset prompts can be set to guide the large language model to better complete the semantic comparison task. For example, prompts such as "determine whether the following two texts are semantically similar" or "calculate the semantic similarity of the following text pairs" can be used. The introduction of prompts helps the model focus on specific comparison tasks, thereby improving the accuracy and reliability of the semantic comparison results. Semantic comparison using a large language model can not only capture explicit semantic relationships between texts but also identify implicit semantic associations in complex contexts. This method overcomes the limitations of traditional comparison methods that rely solely on surface similarity, enabling deeper text matching at a higher level. It is evident that the quantification and analysis of semantic comparison results provide strong support for the accuracy of text comparison. This comparison approach ensures accurate identification and handling of potential content differences and coverage gaps in complex text environments, ultimately providing users with efficient and reliable comparison results.

[0095] S15: Traverse all non-repeating key information and, based on all semantic comparison results, output the text comparison results of the first and second texts to be compared, thus completing the text comparison.

[0096] In this embodiment, to generate a comprehensive and accurate comparison report, it is necessary to traverse all unique key information and perform a comprehensive content comparison between the first and second texts to be compared, ultimately outputting the text comparison results. Specifically, each unique key information and its corresponding text to be compared carries a semantic comparison result. By traversing all unique key information and checking the comparison result between each key and its text to be compared one by one, the coverage between the texts is comprehensively judged. Once the traversal is complete, a text comparison result report is generated. The report clearly lists each unique key information in the first text to be compared, as well as its corresponding matching status. The report highlights any missing information in the text to be compared, helping users quickly locate potential content deviations or omissions and complete the comparison task. Therefore, in this way, the comparison results not only help users verify the consistency of the documents but also provide a clear direction for subsequent rectification and improvement work. The entire process improves the transparency and accuracy of text comparison and enhances the efficiency and quality of compliance management.

[0097] This application efficiently extracts the first key point information of each logical text block and locates its position within the original text by segmenting the first text to be compared and combining preset prompt words with a large language model. This process ensures the comprehensiveness and accuracy of the comparison benchmark, laying the foundation for the comparison task. Simultaneously, semantic segmentation and prompt word guidance effectively reduce interference from redundant information and improve the quality of key information extraction. Secondly, for multiple first key point information within the same positional range, preset key point information processing rules are used to obtain non-repeating key point information and perform quantification. This method not only achieves semantic deduplication of redundant information but also generates first quantified information representing key point information. This quantification transforms the semantic features of the text, facilitating comparison. This step improves the refinement of the comparison data, ensuring that each piece of information is unique and representative in the comparison.

[0098] During the comparison phase, by segmenting the second text to be compared into blocks and generating a comparison vector database, the second quantification information matching the first quantification information can be efficiently retrieved. The extraction of the comparison information supports further semantic comparison, enabling the comparison to focus on the core content between the texts. Next, using a large language model, semantic comparison is performed on non-repeating key point information and its corresponding comparison information to determine the semantic consistency between the two texts. This semantic comparison overcomes the limitations of traditional character matching, not only identifying surface similarities but also capturing deeper semantic connections, thus ensuring the comprehensiveness and accuracy of the comparison results. Finally, by traversing all non-repeating key point information, a final text comparison report is generated based on the comparison results. The report intuitively reflects the differences in content coverage between the first and second texts to be compared, thereby identifying potential omissions or inconsistencies in the text.

[0099] In one embodiment, Figure 2 This is a schematic flowchart illustrating a method for obtaining first quantization information provided in an embodiment of this application. It specifically describes one implementation of step S12 above, which involves obtaining non-repeating keypoint information corresponding to each location range and calculating the first quantization information for each non-repeating keypoint. For example... Figure 2 As shown, based on the above embodiments, it includes:

[0100] S21, filter and obtain multiple first key point information belonging to the same location range, use a preset semantic similarity model to identify and merge similar first key point information to obtain the second key point information corresponding to each location range; wherein, the semantic similarity model is a logistic regression model based on edit distance and vector cosine similarity;

[0101] S22, Based on the location range, the second key point information is spliced ​​together to obtain the non-repeating key point information corresponding to each location range;

[0102] S23, based on the non-repeating key point information, use embedding technology to quantize and obtain the first quantization information of each non-repeating key point information.

[0103] In this embodiment, to improve the accuracy and efficiency of the comparison, it is necessary to filter and integrate multiple first key point information extracted from the same location range. The same location range typically refers to content fragments in the original text that have similar or identical semantic and structural levels. Within the same location range, different key point information may appear in different ways, but their semantic core is often similar or identical. For example, in a regulatory document, "Chapter 3 Risk Management" contains multiple paragraphs, and the key point information in these paragraphs all belong to the location range of "Chapter 3". Similarly, a paragraph may list multiple points regarding data protection, and the key point information of these points all belong to the location range of that paragraph. Likewise, in a contract's "Article 5 Payment Terms," ​​the payment method, deadline, and liability for breach of contract may be detailed, and this information also belongs to a common location range. The specific location range division can be flexibly selected according to the document type of the text to be compared; this is merely an example. Directly including this information in the comparison may lead to computational redundancy and semantic duplication, thus affecting the accuracy of the comparison results. By integrating key information, semantically similar or repetitive key points can be merged within the same location range, resulting in a more refined set of key point information. This approach not only reduces data volume and improves the efficiency of subsequent quantification and comparison, but also avoids misjudgments caused by redundant data. Furthermore, the integrated information more comprehensively reflects the core content within that location range, ensuring semantic context consistency and avoiding semantic fragmentation caused by scattered information.

[0104] A pre-defined semantic similarity model was employed to identify and merge similar first keypoint information to obtain second keypoint information corresponding to each position range. Specifically, the semantic similarity model is a logistic regression model based on edit distance and vector cosine similarity. Edit distance measures the degree of difference between two texts at the character level, reflecting the surface similarity of the texts; vector cosine similarity, by converting the text into a vector representation and calculating the cosine value of the angle between them in vector space, reflects the semantic similarity of the texts. The logistic regression model combines these two features to more accurately determine the semantic similarity between two keypoint information. Through this semantic similarity model, multiple first keypoint information belonging to the same position range are compared, semantically similar keypoint information is identified, and merged. This step effectively eliminates redundant information, ensuring that the keypoint information within each position range is unique and representative, thus obtaining the corresponding second keypoint information. Next, based on the position range, these second keypoint information are concatenated to form non-repeating keypoint information corresponding to each position range. This concatenation method integrates related keypoint information into a whole, facilitating subsequent processing and analysis. At the same time, the spliced ​​non-repeating key point information more comprehensively reflects the core content within that location range.

[0105] Then, for each unique key point, embedding technology is used for quantization to obtain its corresponding first quantized information. Embedding technology maps text to a high-dimensional vector space, allowing the semantic features of the text to be represented in numerical form. Commonly used embedding methods include word vectors (such as word vector generation models and global vector generation models), sentence vectors (such as transformers represented by bidirectional encoders and BERT-based sentence vector models), and embedding service models (such as BigGeneral Embedding, or BGE or General Text Embedding, or GTE). Through this quantization method, the semantic information of the text is transformed into vector data that can be calculated and compared, providing a foundation for subsequent semantic comparison and similarity calculation. In summary, by applying semantic similarity models and embedding technology, efficient filtering, merging, and quantification of key point information are achieved, improving the accuracy and efficiency of text comparison and laying a solid foundation for the smooth progress of subsequent steps.

[0106] In one embodiment, Figure 3 This is a schematic flowchart illustrating the method for obtaining semantic comparison results of non-duplicating key point information and information to be compared, as provided in an embodiment of this application. It specifically explains the use of a large language model in step S31 above to obtain the semantic comparison results. Figure 3 As shown, based on the above embodiments, it includes:

[0107] S31. Based on each set of non-repeating key point information and comparison information, a large language model with preset prompt words is used to calculate and obtain the semantic similarity between the non-repeating key point information and the comparison information.

[0108] S32, if the semantic similarity between the non-repeating key point information and the information to be compared is greater than or equal to the similarity threshold, then it is determined that the information to be compared and the non-repeating key point information express the same meaning.

[0109] S33. If the semantic similarity between the non-repeating key point information and the information to be compared is less than the similarity threshold, then it is determined that the information to be compared and the non-repeating key point information express different meanings.

[0110] In this embodiment, a large language model is used for semantic comparison of each set of non-repeating key information and its corresponding comparison information. During this process, based on the specific task scenario, preset prompts are used to guide the large language model to focus on a specific content, ensuring the accuracy and consistency of the comparison results. The prompts are part of the input to the large language model and aim to clarify the semantic focus of the comparison task. The prompts can be set as "Please judge the semantic similarity between the following two texts" or "Do the following texts express the same semantic meaning?" This guidance method enables the large language model to focus on key information during comparison, eliminating irrelevant content interference and improving the credibility of the comparison results. During the comparison process, the large language model performs semantic analysis on the non-repeating key information and the comparison information. The large language model comprehensively utilizes its language understanding capabilities to calculate the semantic similarity between the two from multiple dimensions (such as lexical semantics, contextual association, syntactic structure, etc.). For example, the similarity calculation results are usually expressed in numerical form, ranging from 0 to 1; the higher the value, the closer the text segments are semantically. If the semantic similarity between a set of non-repeating key information and the information to be compared is greater than or equal to a threshold, then the information to be compared is determined to express the same meaning semantically as the non-repeating key information. This determination means that the text to be compared fully covers the content of the first text to be compared at that information point. Conversely, if the semantic similarity is less than the threshold, then the information to be compared is considered to express a different meaning from the non-repeating key information. This difference may indicate that the text to be compared has a misunderstanding or incomplete coverage at that information point, thus prompting the user to further verify that part of the content. Through this step, this application achieves in-depth semantic analysis in text comparison. The large language model can not only capture explicit semantic similarity, but also identify implicit semantic associations, overcoming the limitations of traditional text comparison methods that rely solely on character matching or surface similarity. Ultimately, this method effectively improves the accuracy and reliability of text comparison.

[0111] Figure 4This is a flowchart illustrating the method for outputting text comparison results provided in an embodiment of this application. It specifically describes one implementation of step S15 described above. Based on the above embodiment, as... Figure 4 As shown, it includes:

[0112] S41, Traverse all unique key point information and obtain the semantic comparison results corresponding to all unique key point information;

[0113] S42, after determining that the information to be compared and the non-duplicated key point information express the same meaning, mark the non-duplicated key point information as covered items and store it in the covered dataset;

[0114] S43, after determining that the information to be compared and the non-duplicated key point information express different meanings, mark the non-duplicated key point information as uncovered items and store them in the uncovered dataset;

[0115] S44. Based on the covered dataset and / or uncovered dataset, output the text comparison results of the first text to be compared and the second text to be compared, and complete the text comparison.

[0116] In this embodiment, by traversing all unique keypoint information and analyzing their semantic matching degree with the information to be compared, text comparison results are generated. During the traversal, based on the semantic comparison results of each set of unique keypoint information, it is determined whether the information to be compared expresses the same meaning semantically. Specifically, if the semantic comparison results show that the semantic similarity between the information to be compared and the unique keypoint information reaches a preset similarity threshold, then it is determined that the two are semantically matched. In this case, the unique keypoint information is marked as a covered item and stored in the covered dataset. This process ensures that the information coverage of each keypoint in the comparison is accurately recorded.

[0117] On the other hand, if the semantic comparison results show that the semantic similarity between the information to be compared and the non-repeating key point information is lower than the similarity threshold, it is considered that the information to be compared fails to semantically cover the non-repeating key point information. In this case, the non-repeating key point information is marked as an uncovered item and stored in the uncovered dataset. Storing uncovered items provides users with clear guidance on content differences, facilitating the quick identification of potential omissions or deviations in the text to be compared. Once the traversal and marking of all non-repeating key point information is completed, a text comparison result report will be generated based on the covered and uncovered datasets. For example, the report content will be presented in a clear and intuitive form, specifically including the following information:

[0118] Covered items: A detailed list of each semantically fully covered, non-repeating key point information and its corresponding comparison information, proving that this part of the content has been fully reflected in the second comparison text.

[0119] Missing items: Lists each non-repeating key point that failed to match semantically, indicating the missing content in the text to be compared, and prompting the user to pay special attention to these contents.

[0120] Ultimately, the text comparison report provides users with comprehensive insights into the content matching and coverage of the two texts. Through clear categorization of covered and uncovered items, users can quickly pinpoint differences in the content, identify potential risks and deficiencies, and take timely corrective action. This step, through systematic analysis and intuitive report output, achieves high efficiency and reliability in text comparison.

[0121] Figure 5 This is a schematic flowchart illustrating a method for constructing a comparison vector database according to an embodiment of this application. Based on the above embodiments, as follows... Figure 5 As shown, it includes:

[0122] S51, obtain multiple second logical text blocks of the second text to be compared, and generate second quantization information corresponding to each second logical text block using a preset embedding technique based on all the second logical text blocks;

[0123] S52, store all the second quantization information into the comparison vector database, and create an index on the second quantization information.

[0124] In this embodiment, to achieve efficient content matching and comparison, the second text to be compared needs to be parsed to obtain multiple corresponding second logical text blocks. After obtaining the second logical text blocks, each second logical text block is quantized. The quantization process is implemented through a preset embedding technique, which can transform text information into a high-dimensional vector representation. Commonly used embedding techniques include, but are not limited to, BGE (Bidirectional General Encoder) and GTE (General Text Encoder) models. These models can capture the deep semantic features of the text and map semantically similar texts to adjacent vector space positions. Specifically, each second logical text block is processed by the embedding model to generate a unique second quantized information, i.e., a high-dimensional semantic vector. The generated second quantized information not only retains the semantic content of the original text but also has a numerical representation, facilitating rapid similarity retrieval and comparison in large-scale data scenarios. To further improve comparison efficiency, all second quantized information is stored in the comparison vector database. To optimize the database query performance, an index structure is established on all second quantized information. The index structure can employ various high-dimensional data indexing techniques, such as KD-trees, Ball trees, or Approximate Nearest Neighbor (ANN) algorithms. These indexing techniques can shorten the time for vector matching and similarity retrieval, ensuring rapid response even with large-scale text data. Through this series of processing steps, the second text to be compared is successfully transformed into a numerical semantic vector set and stored in a vector database. This database provides efficient retrieval support for subsequent comparison tasks, ensuring accuracy and real-time performance during the text comparison process. Ultimately, this combination of embedding and indexing techniques provides the text comparison system with powerful data processing and analysis capabilities, meeting the needs of complex text comparison application scenarios.

[0125] Furthermore, Figure 6 This is a schematic flowchart illustrating a method for retrieving and filtering second quantification information and obtaining comparison information provided in an embodiment of this application. Based on the above embodiments, as... Figure 6 As shown, it includes:

[0126] S61, based on the first quantization information, search the comparison vector database to obtain a set of second quantization information that matches the first quantization information;

[0127] S62, based on the second quantization information in the second quantization information set, obtain the second logical text block corresponding to each second quantization information, and record the second logical text block as the comparison information of non-repeating key points.

[0128] In this embodiment, the retrieval process uses the first quantified information as the query basis and calculates its semantic similarity with each second quantified information in the database. Specifically, cosine similarity, Euclidean distance, or other suitable similarity measures can be used to calculate the proximity of two vectors in the semantic space. The higher the similarity, the closer the two texts are semantically. To improve retrieval efficiency, the index structure of the database is used to quickly locate matching items. After the retrieval is completed, a set of second quantified information matching the first quantified information is returned. This set contains all second quantified information that is highly similar to the first quantified information semantically, providing candidate data for subsequent comparisons. Next, based on the set of second quantified information, the second logical text block corresponding to each second quantified information is searched one by one. Each quantified information corresponds one-to-one with the original text block. Through this association, the numerical comparison result can be restored to the specific text content. The corresponding second logical text block represents the part of the text to be compared that may match the non-repeating key point information in the first text to be compared. To ensure the comprehensiveness and accuracy of the comparison, each found second logical text block is recorded as the comparison information of non-repeating key points. This information storage method not only preserves important semantic relationships in text comparison but also lays the foundation for subsequent semantic comparison and final report generation. In summary, this retrieval and association process efficiently extracts comparison content related to key information from large-scale text data. This method combines efficient vectorized retrieval with semantic relationships within text content, improving the accuracy and coverage of comparison results and providing reliable technical support for comparison needs in complex text environments.

[0129] Figure 7 This is a flowchart illustrating a method for determining the location range of the first key point information provided in an embodiment of this application. It is a specific explanation of how the location range of the first key point information is determined in step S11 above. Based on the above embodiment, as... Figure 7 As shown, it includes:

[0130] S71, based on the preset prompt words, according to the first logical text block, a large language model is used to obtain the first key point information associated with the preset prompt words;

[0131] S72, Traverse the first logical text block set and obtain the first key point information corresponding to each first logical text block;

[0132] S73, based on the preset text structure of the first text to be compared, obtain the location information of the first key point information, and determine the location range of the first key point information in the first text to be compared based on the location information.

[0133] In this embodiment, to extract key information from the first text to be compared, each first logical text block needs to be analyzed based on preset prompts. These prompts can be flexibly customized according to specific application scenarios, such as "extract the core clauses of the following text" or "list the main policy points in the following paragraphs." During the processing of each first logical text block, a large language model is invoked. The large language model utilizes its deep semantic understanding capabilities, combined with the guidance of prompts, to extract the first key information associated with the prompts from complex text. This key information can effectively reflect the main intent or important clauses of the text. To ensure that the key information of all text blocks is covered, the set of first logical text blocks is traversed, and each text block is processed one by one. Through this traversal operation, the corresponding first key information can be extracted from each first logical text block, ultimately forming a complete set of key information covering the entire first text to be compared. After extracting the key information, it is also necessary to locate each piece of first key information according to the preset text structure of the first text to be compared. The preset text structure refers to the inherent hierarchical organization of chapters, clauses, paragraphs, etc., in the first text to be compared. This structural information provides the foundation for locating key information. By analyzing the position of each first key piece of information in the original text, its specific location information can be obtained, such as the chapter number, paragraph number, or character index. Using this location information, the position range of each first key piece of information in the first text to be compared can be further determined (i.e., the chapter, clause, or paragraph to which the first key piece of information belongs). Through the above steps, the function of extracting and locating key information from the first text to be compared is realized. This method fully utilizes the powerful semantic understanding capabilities of large language models and the organizational advantages of pre-defined text structures, providing a semantic foundation and location support for subsequent comparison tasks.

[0134] Figure 8 This is a schematic flowchart illustrating a method for obtaining logical text blocks provided in an embodiment of this application. Based on the above embodiments, as follows... Figure 8 As shown, it includes:

[0135] S81, parse and preprocess the file to be compared to obtain the text information of the file to be compared;

[0136] S82, based on the text information, use a trained segmentation model to obtain multiple logical text blocks of the text information; wherein, the segmentation model includes at least one of the BERT model and the SemanticChunker model.

[0137] In this embodiment, to ensure the efficiency and accuracy of the comparison process, the files to be compared first need to be parsed and preprocessed. This typically includes removing irrelevant information, standardizing text format, and processing special symbols and noisy data in the text. Specifically, the files to be compared may contain various forms of unstructured data, such as titles, annotations, and chart descriptions. Through parsing and preprocessing, this data can be filtered or transformed into a structured plain text format, ensuring the consistency and integrity of the text data. After preprocessing, based on the preprocessed text information, a trained chunking model is used to divide the text into multiple logical text blocks. The BERT model (i.e., a bidirectional encoder-represented transformer) is a contextual embedding model that captures deep semantic relationships between words by bidirectionally encoding each word in a sentence. During chunking, the BERT model can automatically determine the breakpoint positions of the text based on the contextual semantics of the sentence, ensuring that each logical text block is semantically independent and complete. This is particularly important for processing complex, non-linear text structures. The SemanticChunker model focuses on chunking text through semantic analysis. This model can divide related sentences or paragraphs into logical units based on the semantic features of the text. This segmentation method is particularly suitable for long texts or documents with a clear structure, such as legal documents or contract terms. The SemanticChunker model ensures that the segmented text blocks are highly semantically and logically related, thus providing more accurate semantic units for subsequent comparisons. It should be noted that the most suitable segmentation method can be flexibly selected based on the text content and structural features.

[0138] Based on the above embodiments, multiple first logical text blocks corresponding to the first text to be compared are obtained, including:

[0139] S811, parse and preprocess the first file to be compared to obtain the text information of the first text to be compared;

[0140] S821, Based on the text information of the first text to be compared, the trained BERT model is used to obtain multiple first logical text blocks of the first text to be compared.

[0141] In this embodiment, the parsing process aims to transform the original format of the first file to be compared into standardized text information. The file may exist in various forms, including PDF, Word documents, or other text formats. During parsing, unstructured files are converted into machine-readable text formats, for example, extracting text content from PDFs or scanned documents and converting it into plain text. Next, the core content of the file is identified and extracted, removing semantically irrelevant content such as headers, footers, page numbers, and appendices. Subsequently, the character encoding of the text is unified to a standard format (such as UTF-8) to avoid text parsing errors caused by inconsistent encoding. Finally, chapters, clauses, paragraphs, and other content in the text are marked to provide an organizational framework for subsequent logical text block generation. After parsing and preprocessing, based on the obtained text information of the first text to be compared, a trained BERT model is used to perform semantic segmentation of the text. The BERT model (Bidirectional Encoder Representations from Transformers) is a deep pre-trained language model capable of capturing semantic breakpoints in text through bidirectional contextual understanding. Specifically, the BERT model analyzes text sentence by sentence or paragraph by paragraph, leveraging its semantic encoding capabilities to divide content with similar contextual meanings into independent logical text blocks. This segmentation approach not only considers the surface grammatical structure but also delves into the semantic dependencies within the text, ensuring the integrity of each logical text block in terms of content expression. For example, when a change in topic or content is detected in a paragraph, a new logical text block is generated at that location. Compared to traditional fixed segmentation by paragraph or chapter, this dynamic segmentation method can more accurately capture semantic changes in the text, providing high-quality data input for subsequent key point extraction and comparison tasks. Through the above processing, multiple semantically independent first logical text blocks can be generated from the first text to be compared. These text blocks not only lay a solid foundation for key point information extraction but also provide higher semantic analysis accuracy and operability for the entire text comparison process.

[0142] Simultaneously, multiple second logical text blocks corresponding to the second text to be compared are obtained, including:

[0143] S812 parses and preprocesses the second file to be compared to obtain the text information of the second file to be compared.

[0144] S822, Based on the text information of the second file to be compared, the trained BERT model is used to obtain multiple second logical text blocks of the second file to be compared.

[0145] In this embodiment, the steps for obtaining the second logical text block are the same as those for obtaining the first logical text block, and will not be repeated here.

[0146] like Figures 9 to 11 As shown, in one specific embodiment, this embodiment provides a text comparison method based on a large language model, used to compare a first text to be compared (i.e., regulatory policies or guidance documents issued by the superior regulatory authority, hereinafter referred to as "regulatory documents") and a second text to be compared (i.e., internal guidance documents obtained by interpreting and splitting the regulatory documents, hereinafter referred to as "internal guidance documents"), so as to ensure that the internal guidance documents fully cover regulatory requirements.

[0147] First, obtain regulatory documents and internal guidance documents. Parse these documents to extract their text information. The parsing process includes converting the documents to a processable plain text format, removing irrelevant formatting and tags, and ensuring the integrity and accuracy of the text content. For files in formats such as PDF and Word, appropriate parsing tools or libraries (such as PDFBox or Apache POI) can be used for text extraction.

[0148] Next, based on the inherent semantic information of the text, the regulatory documents and internal guidance documents are segmented into multiple first and second logical text blocks. Segmentation methods can employ embedding-based methods, pre-trained segmentation models, or large language model-based methods. For example, the SemanticChunker tool provided by LangChain can be used to segment the text based on semantic information. Alternatively, a trained BERT model can be used based on SeqModel to encode multiple sentences, considering contextual information and capturing semantic dependencies between sentences to determine sentence segmentation points and perform semantic segmentation of the document. Through these methods, multiple logical text blocks with independent semantics are ultimately obtained.

[0149] For each logical text block, it is associated with the chapter or clause information it belongs to, forming a text block set W = {w1, w2, w3, ..., wn}, where each item wi = {Ti, Si}, Ti represents the current chapter or clause information, and Si represents the logical text block. This yields the set of regulatory documents WS and the set of internal guidance documents WG.

[0150] Then, the logical text block set WG of the internal guidance document is vectorized and stored. All Si (second logical text blocks) in the WG set are sent to the embedding service and processed using a preset embedding technique (such as the general embedding model BGE or the general text embedding model GTE) to obtain the second quantized information Vi corresponding to each Si, i.e., a high-dimensional semantic vector. Ti, Si, and Vi are combined to form a complete data item Di={Ti, Si, Vi}, and stored in the vector database to be compared. Simultaneously, an index is built on all Vi to improve the efficiency of subsequent retrieval. The index can use common high-dimensional vector index structures, such as BallTree, KD-Tree, or Approximate Nearest Neighbor (ANN) algorithm.

[0151] Next, the logical text block set WS of the regulatory document is analyzed to extract the first key point information. Each Si (first logical text block) in WS, combined with preset prompt words, is input into a large language model for processing. The large language model can use the open-source Llama model or Yi model, or a model fine-tuned according to specific domain data. The large language model extracts the first key point information Pi from each Si according to the requirements of the prompt words. Through the relationship of the text block set, Pi corresponds to a chapter or clause information Ti. A chapter or clause information usually corresponds to multiple Pis. After sorting, a key point information set PT={PT1, PT2, ..., PTn} with chapter or clause information as the set feature is obtained, where PTi={Pi1, Pi2, ..., Pin}.

[0152] To eliminate duplicate or similar first keypoint information within the same location range, each PTi is merged. A pre-defined semantic similarity model is used to identify and merge similar first keypoint information to obtain the second keypoint information corresponding to each location range. The semantic similarity model is a logistic regression model (LR model) based on edit distance and vector cosine similarity. Edit distance measures the difference between two texts at the character level, while vector cosine similarity measures the similarity of texts in the semantic vector space. The logistic regression model combines these features and, after training, can accurately determine the semantic similarity between two text segments. Through this model, similar items in PTi are merged to obtain the second keypoint information set PTi'.

[0153] Then, based on the location range, the second keypoint information Pi' in each PTi' is concatenated to obtain the non-repeating keypoint information Si' corresponding to each location range. Next, embedding technology is used to quantize each non-repeating keypoint information Si' to obtain the first quantized information Vi' of each non-repeating keypoint information. Similarly, a preset embedding model (such as BGE or GTE) is used to transform Si' into a high-dimensional semantic vector Vi' for subsequent vector retrieval.

[0154] Based on this, for each Vi', the matching second quantization information Vi is retrieved from the comparison vector database. The retrieval process uses vector similarity calculation, typically using cosine similarity or Euclidean distance to measure the similarity between Vi' and Vi. Because an index is built on Vi, the retrieval process can be completed efficiently. The obtained matching results form a candidate set Ci, which contains the second quantization information Vi that is similar to Vi'.

[0155] Then, the second logical text block Si corresponding to each Vi in the candidate set Ci is obtained as the comparison information. Each unique key point information Si' and the corresponding comparison information Si, combined with preset prompt words, are input into the large language model to obtain the semantic comparison results of the unique key point information and the comparison information. The prompt words can be set as: "Please judge whether the following two texts express the same meaning semantically" or "Please compare whether the content of the following texts is consistent".

[0156] The large language model performs semantic analysis on the input text pairs and calculates their semantic similarity. If the semantic similarity is greater than or equal to a preset similarity threshold (e.g., 0.8), it is determined that the information to be compared and the non-repeating key point information express the same meaning. The non-repeating key point information Pi' and the corresponding chapter information Ti are stored in the result set R1, indicating a covered item. If the semantic similarity is less than the similarity threshold, it is determined that the two express different meanings. Pi' and Ti are stored in the result set R2, indicating an uncovered item.

[0157] Repeat the above comparison process, iterating through all unique key information points, to obtain the semantic comparison results corresponding to all unique key information points. Finally, based on result sets R1 and R2, a text comparison result report is generated. The report lists the covered items in result set R1, demonstrating the regulatory requirements already covered in the internal guidance document, providing the corresponding sections and text blocks for review and confirmation. For the uncovered items in result set R2, each uncovered regulatory requirement is displayed, highlighting areas requiring attention and supplementation. These uncovered items may represent compliance risks present in the internal guidance document, requiring timely handling by relevant departments.

[0158] Through the specific embodiments described above, the method of this application achieves automated and accurate text comparison of regulatory documents and internal guidance documents. Employing large language models and embedding technology, it can deeply understand the semantics of the text, accurately extract key information, and perform efficient semantic comparison. Utilizing vector databases and index structures improves the efficiency of retrieval and comparison. The generated text comparison report clearly demonstrates the coverage of regulatory requirements by the internal guidance documents, providing strong technical support for content review and internal management.

[0159] Figure 12 This is a schematic diagram of the text comparison device provided in an embodiment of this application. Figure 12 As shown, the text comparison device 12 includes:

[0160] The key point information acquisition module 121 is used to acquire multiple first logical text blocks corresponding to the first text to be compared, and according to the preset prompt words, adopts a large language model to acquire the first key point information corresponding to each first logical text block, and determines the position range of the first key point information in the first text to be compared.

[0161] The information quantization module 122 is used to obtain non-repeating key point information corresponding to each location range based on multiple first key point information within the same location range, according to preset key point information processing rules, and to calculate the first quantization information of each non-repeating key point information.

[0162] The information retrieval module 123 is used to obtain multiple second logical text blocks corresponding to the second text to be compared, and based on the comparison vector database generated from the second logical text blocks, retrieve the second quantization information that matches the first quantization information, and record the comparison information that the second logical text blocks corresponding to the second quantization information are non-repeating key points.

[0163] The information comparison module 124 is used to obtain the semantic comparison results of the non-repeating key point information and the comparison information to be compared using a large language model based on the non-repeating key point information and the comparison information to be compared.

[0164] The result output module 125 is used to traverse all non-repeating key point information and, based on all semantic comparison results, output the text comparison results of the first and second texts to be compared, thus completing the text comparison.

[0165] The text comparison device provided in this embodiment can execute the text comparison method of the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0166] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 13 As shown, the electronic device 13 includes at least one processor 131 and a memory 132. The electronic device 13 also includes a communication component 133. The processor 131, the memory 132, and the communication component 133 are connected via a bus 134.

[0167] In the specific implementation process, at least one processor 131 executes computer execution instructions stored in memory 132, causing at least one processor 131 to execute the entity question-and-answer generation method executed on the electronic device side as described above.

[0168] The specific implementation process of processor 131 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0169] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0170] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0171] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0172] The above description of the functions implemented by electronic devices and main control devices has introduced the solutions provided by the embodiments of the present invention. It is understood that, in order to implement the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0173] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above method.

[0174] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0175] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0176] This application also provides a computer program product, comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the scheme provided in any of the above embodiments.

[0177] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0178] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0179] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0181] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0183] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0184] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0185] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0186] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A text comparison method based on a large language model, characterized in that, include: Multiple first logical text blocks corresponding to the first text to be compared are obtained, and according to the preset prompt words, a large language model is used to obtain the first key point information corresponding to each first logical text block, and the position range of the first key point information in the first text to be compared is determined. Based on multiple first key point information within the same location range, according to the preset key point information processing rules, obtain non-repeating key point information corresponding to each location range, and calculate the first quantitative information of each non-repeating key point information. Obtain multiple second logical text blocks corresponding to the second text to be compared, and based on the comparison vector database generated from the second logical text blocks, retrieve second quantization information that matches the first quantization information, and record the comparison information where the second logical text block corresponding to the second quantization information is the non-repeating key point; Based on the non-repeating key point information and the comparison information of the non-repeating key points, a large language model is used to obtain the semantic comparison results of the non-repeating key point information and the comparison information. Iterate through all the non-repeating key point information and, based on all the semantic comparison results, output the text comparison results of the first text to be compared and the second text to be compared, thus completing the text comparison.

2. The method according to claim 1, characterized in that, Based on multiple first key point information within the same location range, and according to preset key point information processing rules, non-repeating key point information corresponding to each location range is obtained, and first quantized information of each non-repeating key point information is calculated, including: Filter and obtain multiple first key point information belonging to the same location range, use a preset semantic similarity model to identify and merge similar first key point information to obtain the second key point information corresponding to each location range; The semantic similarity model is a logistic regression model based on edit distance and vector cosine similarity. Based on the location range, the second key point information is spliced ​​together to obtain non-repeating key point information corresponding to each location range; Based on the non-repeating key point information, an embedding technique is used to quantize and obtain the first quantization information of each non-repeating key point information.

3. The method according to claim 1, characterized in that, The step of obtaining the semantic comparison results of the non-repeating key point information and the comparison information to be compared using a large language model based on the non-repeating key point information and the comparison information to be compared includes: Based on the non-repeating key point information and the information to be compared in each group, a large language model with preset prompt words is used to calculate and obtain the semantic similarity between the non-repeating key point information and the information to be compared. If the semantic similarity between the non-repeating key point information and the information to be compared is greater than or equal to a similarity threshold, then it is determined that the information to be compared and the non-repeating key point information express the same meaning; or, If the semantic similarity between the non-repeating key point information and the information to be compared is less than a similarity threshold, then it is determined that the information to be compared and the non-repeating key point information express different meanings.

4. The method according to claim 3, characterized in that, The step of traversing all the non-repeating key point information and, based on all the semantic comparison results, outputting the text comparison results of the first text to be compared and the second text to be compared, includes: Traverse all the non-repeating key point information and obtain the semantic comparison results corresponding to all the non-repeating key point information. After determining that the information to be compared and the non-repeating key point information express the same meaning, the non-repeating key point information is marked as a covered item and stored in the covered dataset; After determining that the information to be compared and the non-repeating key point information express different meanings, the non-repeating key point information is marked as an uncovered item and stored in the uncovered dataset; Based on the covered dataset and / or uncovered dataset, output the text comparison results of the first text to be compared and the second text to be compared, and complete the text comparison.

5. The method according to claim 1, characterized in that, The construction of the vector database to be compared includes: Obtain multiple second logical text blocks of the second text to be compared, and generate second quantization information corresponding to each second logical text block using a preset embedding technique based on all the second logical text blocks; All the second quantization information is stored in the comparison vector database, and an index is created on the second quantization information; Based on the comparison vector database generated from the second logical text block, second quantization information matching the first quantization information is retrieved, and comparison information recording the second logical text block corresponding to the second quantization information as the non-repeating key point is recorded, including: Based on the first quantization information, a search is performed in the comparison vector database to obtain a second quantization information set that matches the first quantization information. Based on the second quantization information in the second quantization information set, obtain the second logical text block corresponding to each second quantization information, and record the second logical text block as the comparison information of the non-repeating key points.

6. The method according to any one of claims 1-3, characterized in that, The step of obtaining the first key point information corresponding to each first logical text block based on preset prompt words and using a large language model, and determining the position range of the first key point information in the first text to be compared, includes: Based on the preset prompt words, according to the first logical text block, the large language model is used to obtain the first key point information associated with the preset prompt words; Traverse the first set of logical text blocks and obtain the first key point information corresponding to each first logical text block; Based on the preset text structure of the first text to be compared, the location information of the first key point information is obtained, and based on the location information, the location range of the first key point information in the first text to be compared is determined.

7. The method according to claim 1 or 5, characterized in that, The method further includes: The file to be compared is parsed and preprocessed to obtain the text information of the file to be compared. Based on the text information, a trained block model is used to obtain multiple logical text blocks of the text information; The block model includes at least one of the BERT model and the SemanticChunker model; The step of obtaining multiple first logical text blocks corresponding to the first text to be compared includes: The first file to be compared is parsed and preprocessed to obtain the text information of the first text to be compared. Based on the text information of the first text to be compared, a trained BERT model is used to obtain multiple first logical text blocks of the first text to be compared. Then, obtain multiple second logical text blocks corresponding to the second text to be compared, including: The second file to be compared is parsed and preprocessed to obtain the text information of the second file to be compared. Based on the text information of the second file to be compared, a trained BERT model is used to obtain multiple second logical text blocks of the second file to be compared.

8. A text comparison device, characterized in that, include: The key point information acquisition module is used to acquire multiple first logical text blocks corresponding to the first text to be compared, and according to the preset prompt words, adopts a large language model to acquire the first key point information corresponding to each first logical text block, and determines the position range of the first key point information in the first text to be compared. The information quantification module is used to obtain non-repeating key point information corresponding to each location range based on multiple first key point information within the same location range, according to preset key point information processing rules, and to calculate the first quantification information of each non-repeating key point information. The information retrieval module is used to obtain multiple second logical text blocks corresponding to the second text to be compared, and based on the comparison vector database generated from the second logical text blocks, retrieve second quantization information that matches the first quantization information, and record the comparison information that the second logical text block corresponding to the second quantization information is the non-repeating key point. The information comparison module is used to obtain the semantic comparison results of the non-repeating key point information and the comparison information to be compared using a large language model, based on the non-repeating key point information and the comparison information to be compared. The result output module is used to traverse all the non-repeating key point information and, based on all the semantic comparison results, output the text comparison results of the first text to be compared and the second text to be compared, thus completing the text comparison.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images