Large model-based long text processing method and device, electronic equipment and medium

Abstract

The application discloses a long text processing method and device based on a large model, electronic equipment and a medium, and relates to the technical field of text processing. The application solves the technical problem that there is a lack of targeted processing rules, and the correction of block overlap content and global integration lack systematic methods, and information inconsistency or omission is prone to occur. The application classifies text length and type, matches a differentiated block strategy, ensures semantic integrity, and improves processing efficiency. The application covers five conflict types, including facts, logic, semantics, reference and scope, combines professional tools to realize multi-dimensional conflict accurate positioning, avoids the limitations of single-dimensional analysis, designs differentiated processing rules for different conflict types, and ensures the accuracy of conflict correction. Meanwhile, through overlapping content correction and progressive integration, global information consistency and integrity are realized. The integrated results are verified through a secondary processing mechanism, further ensuring the consistency and integrity of long text processing.

Description

Technical Field

[0001] This invention relates to the field of text processing technology, specifically to long text processing methods, apparatus, electronic devices, and media based on large models. Background Technology

[0002] With the development of the information age, the number of long texts is growing exponentially. These texts are often characterized by their length, complex structure, and large differences in semantic density, posing challenges to efficient processing.

[0003] According to patent application CN120087479A, a long text processing method, apparatus, electronic device, computer-readable storage medium, and computer program product based on a large model are disclosed. The method involves: acquiring a long text to be processed; extracting key points from the long text to obtain multiple key information points; determining the relationships between the multiple key information points; responding to a received query, determining a first key information point matching the query and a second key information point associated with the first key information point based on the query and the relationships, wherein the query is used to obtain corresponding response information based on the long text; and generating response information for responding to the query based on the first and second key information points using a first large model.

[0004] However, in existing technologies, long text processing often adopts a fixed segmentation strategy without considering differences in text types; conflict analysis is often limited to a single dimension and lacks a systematic approach to multi-dimensional conflicts such as logic, semantics, and pronoun reference; and the mechanism for correcting and integrating overlapping content after segmentation is imperfect, which can easily lead to information fragmentation or missing conflicts, making it difficult to meet the needs of in-depth understanding and accurate processing of long texts. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a long text processing method, apparatus, electronic device, and medium based on a large model. It solves the problems of lacking targeted processing rules and lacking a systematic method for correcting overlapping content in blocks and for global integration, which easily leads to inconsistencies or omissions in information.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a long text processing method based on a large model, which specifically includes the following steps:

[0007] S1. Obtain long text information, classify it according to the comparison results of the text length with preset values ​​K1 and K2, and divide the long text into blocks for different categories to generate block information.

[0008] S2. Label the obtained blocks, perform block conflict analysis on the block information, and if there is a block conflict, obtain the corresponding conflict and generate a conflict analysis signal; otherwise, do not process it.

[0009] S3. Process the generated conflict analysis signal to obtain the corresponding conflict. Process the conflict according to different types, including factual conflict, logical conflict, semantic conflict, referential conflict and scope conflict, and obtain the block processing result.

[0010] S4. Obtain the block processing results and label them. Perform overlap judgment on the block processing results. If there is overlap, generate an overlap analysis signal and process it to generate a correction result. Otherwise, generate a result integration signal. Progressively integrate all block processing results to obtain an integrated result, and re-input the integrated result to generate a secondary processing result.

[0011] As a further aspect of the present invention, in step S1, the text length is classified according to the comparison results with preset values ​​K1 and K2. If the text length > K2, it is classified as extra-large text; if K1 < text length ≤ K2, it is classified as normal length text; if the text length ≤ K1, it is classified as normal long text.

[0012] For normal long texts, they are divided into blocks of fixed length; for texts of general length, they are divided into blocks through semantic awareness, that is, by using a model to calculate the similarity between sentences / paragraphs and merging semantically similar units; for very large texts, they are divided into blocks according to structure, that is, by cutting them into levels based on text format.

[0013] As a further aspect of the present invention, the specific method for performing block conflict analysis on the block information is as follows:

[0014] The obtained blocks are labeled as i, and i = 1, 2, ..., j, where j represents the number of blocks. Then, block conflict analysis is performed on block information i. The block conflict analysis is specifically performed separately from factual conflict, logical conflict, semantic conflict, referential conflict and scope conflict. If block information i has a block conflict, the corresponding conflict is obtained and a conflict analysis signal is generated. Otherwise, if block information i does not have a block conflict, it is not processed.

[0015] As a further aspect of the present invention, the specific method for processing different conflicts to obtain the block processing result is as follows:

[0016] Analyze the cases where the conflict is a factual conflict, locate the core entity and corresponding attribute of the conflict, determine whether it is an irreconcilable absolute conflict based on the attribute type, extract the entity-attribute-value triple, distinguish between reconcilable and irreconcilable conflicts, for reconcilable conflicts, first check the original source of the long text to see if there are description differences due to different contexts, if so, correct it, otherwise introduce external authoritative sources to verify and correct, and obtain factual conflict analysis information.

[0017] Analyze the logical conflicts by identifying logical connectors and logical subjects from the segmented information, determine the type of logical conflict, identify causal relationships, and check whether any constraints for the logic to hold are missing from the segmented information. If any are missing, complete them with logical expressions; if none are missing, determine the cause of the conflict through large-scale model reasoning and correct it to obtain logical conflict analysis information.

[0018] As a further aspect of the present invention, the specific method for processing different conflicts to obtain the block processing result is as follows:

[0019] The semantic conflict is analyzed, and the semantic dimension of the conflict is located from the segmented information. The semantic dimension includes same-dimensional conflict and different-dimensional conflict. If it is same-dimensional, the reference standard is identified and corrected to obtain semantic conflict analysis information.

[0020] The system analyzes the conflict of references, extracts reference markers and candidate entities from the blocks, and determines the reference based on the principle of proximity, semantic association, and global frequency. For conflicting references, it prioritizes the use of entities that are clearly marked in the original text or have high-frequency associations, and marks the reference object and the basis to generate reference conflict analysis information.

[0021] For cases where the conflict is a scope conflict, analyze the scope elements and boundary descriptions, verify the inclusion relationship using a hierarchical database, and annotate the scope hierarchy and boundaries to generate scope conflict analysis information.

[0022] As a further aspect of the present invention, the specific method for determining the overlap of the block processing results is as follows:

[0023] Obtain the block processing results and label them as Gi. At the same time, perform overlap judgment on the block processing results Gi. If the block processing results overlap, generate an overlap analysis signal; otherwise, if there is no overlap, generate a result integration signal.

[0024] The generated overlap analysis signal is processed as follows: the corresponding overlap result is obtained, key elements are extracted from the overlap content, the processing results of the block on the overlap area are compared, the conflict type is determined, the complete context of the overlap area in the original text is checked, the same content in the overlap content is obtained, and the unique content of the non-overlapping area of ​​the block is retained, and the correction result is generated.

[0025] The long text processing device based on a large model includes a text classification module, a block processing module, a conflict analysis module, a conflict resolution module, an overlap processing module, and a result integration module.

[0026] The text classification module acquires long texts and, by comparing them with preset values ​​K1 and K2, classifies them into extra-large texts, normal-length texts, and normal-length texts.

[0027] The chunking module divides normal long texts into chunks of fixed length; texts of average length are chunked using Sentence-BERT based on semantic similarity; and very large texts are chunked according to the chapter-section-subsection hierarchy based on format.

[0028] Conflict Analysis Module: Analyzes the segmented information for factual, logical, semantic, referential, and scope conflicts. If there is a conflict, a conflict analysis signal is generated; if there is no conflict, a segmented processing signal is generated.

[0029] In the conflict resolution module, factual conflicts are extracted into triples using tools, and reconciliation is determined and corrected based on attribute type; logical conflicts are identified using tools, and constraints are completed or large model reasoning is corrected; semantic conflicts are located in dimensions, and same-dimensional conflicts are corrected according to reference standards; referential conflicts are determined and labeled based on proximity, semantic association, and global frequency; and scope conflicts are extracted into elements, and the hierarchy and exceptions are verified and labeled using a hierarchical database.

[0030] The overlap processing module determines whether the block results overlap. If there is overlap, it extracts elements, compares and corrects them, and retains the same content and the unique content of the non-overlapping areas. If there is no overlap, it generates an integration signal.

[0031] The results integration module progressively integrates the results of the segmented processing, generates the integrated result, and performs secondary input processing.

[0032] A long text processing electronic device based on a large model, comprising at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a long text processing method based on a large model.

[0033] A long text processing medium based on a large model is used to store computer programs. When the computer programs are executed, they are used to implement long text processing methods based on the large model.

[0034] This invention provides a method, apparatus, electronic device, and medium for processing long texts based on large models. Compared with existing technologies, it has the following advantages:

[0035] This invention employs a dual classification system based on text length and type, matching differentiated segmentation strategies to ensure semantic integrity while improving processing efficiency. It covers five major conflict types: factual, logical, semantic, referential, and scope. Combined with specialized tools, it achieves precise multi-dimensional conflict localization, avoiding the limitations of single-dimensional analysis. Differentiated processing rules are designed for different conflict types to ensure accurate conflict correction. Simultaneously, through overlapping content correction and progressive integration, it achieves global information consistency and integrity, resolving the problem of information fragmentation after segmentation. A secondary processing mechanism verifies the integration results, further ensuring the consistency and integrity of long text processing and providing high-quality data support for subsequent applications. Attached Figure Description

[0036] Figure 1 This is a diagram illustrating the steps and methods of the present invention;

[0037] Figure 2 This is a block diagram of the device of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Example 1

[0040] Please see Figure 1 This application provides a long text processing method based on a large model, which specifically includes the following steps:

[0041] S1. Obtain long text information and classify it according to its text length. Specifically, compare the text length with preset values ​​K1 and K2. If the text length > K2, classify it as extra-large text. If K1 < text length ≤ K2, classify it as normal length text. If the text length ≤ K1, classify it as normal long text. Then, divide the long text into blocks according to different classifications.

[0042] For normal long texts obtained from classification (such as chat logs and logs), the text is divided into blocks of fixed length (e.g., every 2k tokens) to generate block information. For texts of general length obtained from classification (papers, reports, novel chapters), the text is divided into blocks using semantic awareness (using models (Sentence-BERT, GPT-3.5) to calculate the similarity between sentences / paragraphs and merge semantically similar units) to generate block information. For very large texts obtained from classification (such as e-books, legal contracts, and technical manuals), the text is divided into blocks according to structure, specifically based on text format (e.g., Markdown headings, HTML tags, PDF page numbers), such as cutting according to the "chapter-section-subsection" hierarchy to generate block information.

[0043] S2. Label the obtained blocks as i, where i = 1, 2, ..., j, and j represents the number of blocks. Then, perform block conflict analysis on block information i. The block conflict analysis is specifically performed separately on factual conflicts, logical conflicts, semantic conflicts, referential conflicts, and scope conflicts. If block information i has block conflicts, the corresponding conflicts are obtained and a conflict analysis signal is generated. Otherwise, if block information i does not have block conflicts, no processing is performed.

[0044] S3. Process the generated conflict analysis signal and obtain the corresponding conflict. Then, process the conflict according to different conflicts to obtain the block processing result.

[0045] The system analyzes cases where the conflict is a factual conflict, locates the core entity and corresponding attribute of the conflict from the segmented information, determines whether the conflict is an irreconcilable absolute conflict based on the attribute type (absolute attribute / relative attribute), and extracts entity-attribute-value triples using spaCy and HanLP. Specific absolute attributes include time, unique identifier, and precise value, while relative attributes include range and fuzzy value. The system also generates reconcilable and irreconcilable conflicts.

[0046] For reconcilable conflicts, first verify the original source of the long text and check whether the difference in description is due to different contexts. If it is a difference in description, then correct it. If it is not a difference in description, then introduce external authoritative sources (such as databases and official documents) to verify and correct it, and obtain factual conflict analysis information.

[0047] The analysis identifies logical conflicts by distinguishing logical connectors (such as "because-therefore," "if-then," "although-but") and logical subjects (cause A, consequence B) from the segmented information. The types of logical conflicts are then determined, including explicit logical conflicts (directly contradictory connectors, such as "because" and "although") and implicit logical conflicts (no connectors, but contradictions arise through semantic deduction). Specifically, Stanford CoreNLP and TextGraphs are used to identify causal / conditional relationships. Next, it is determined whether any constraints for the logic to hold are missing from the segmented information. If omissions exist, logical expressions are used to complete the analysis. If no omissions are found, the cause of the conflict is determined through reasoning using a large model, and corrections are made to obtain the logical conflict analysis information.

[0048] For cases where the conflict is semantic, we analyze the semantic dimension of the conflict from the segmented information. Specifically, this includes same-dimensional conflict (the conflict is described in opposition within the same dimension) and different-dimensional conflict (the conflict is described in different dimensions). If the conflict is within the same dimension, we identify the reference standard and correct it to obtain semantic conflict analysis information.

[0049] The system analyzes the conflict of references, extracts reference markers and candidate entities from the blocks, and determines the reference based on the principle of proximity, semantic association, and global frequency. For conflicting references, it prioritizes the use of entities that are clearly marked in the original text or have high-frequency associations, and marks them with "reference object + basis" to generate reference conflict analysis information.

[0050] For conflicts that are scope conflicts, the scope elements (subject, region, time) and boundary descriptions are extracted. The inclusion relationship is verified using a hierarchical database (such as administrative divisions and organizational structures). It is checked whether the conflict is caused by exceptions or changes in time / scenario, and the scope level + exception / boundary is marked to generate scope conflict analysis information.

[0051] S4. Obtain the block processing results and label them as Gi. At the same time, perform overlap judgment on the block processing results Gi. If the block processing results overlap, generate an overlap analysis signal; otherwise, if there is no overlap, generate a result integration signal.

[0052] The generated overlap analysis signals are processed to obtain the corresponding overlap results, and then corrected. The specific correction process involves extracting key elements from the overlapping content (such as paragraphs and sentences), including entities and attributes, logical relationships, referents, and scope boundaries. The processing results of the overlapping areas in the blocks are compared to determine the conflict type. The complete context of the overlapping area in the original text is checked to see if the differences are caused by "time changes" (such as progress being updated from 70% to 80%) or "perspective differences" (such as descriptions at different stages). At the same time, the same content in the overlapping content is obtained, and the unique content of the non-overlapping areas of the blocks is retained to generate the correction results.

[0053] Next, the correction results are obtained, and all the block processing results are progressively integrated to obtain the integrated result. The block processing results here include the correction results. At the same time, the integrated result is re-inputted to generate the secondary processing result.

[0054] The block results are sequentially labeled as Gi, and metadata such as the start and end positions and core topics of each block are recorded.

[0055] If the overlapping content of two adjacent blocks (such as Gi and Gi+1) accounts for 10%-30% of the length of a single block (which can be adjusted according to the text type, such as 20%-30% for logic-intensive text), and the overlapping part contains complete semantic units (such as complete sentences or continuous paragraphs), then it is determined to be "effective overlap" and an overlap analysis signal is generated.

[0056] If the overlap ratio is less than 10% or consists only of scattered words (without complete semantics), it is considered "no effective overlap" and the result integration signal is directly generated.

[0057] Four core elements are extracted from the overlapping content, including entities and attributes, logical relationships, referents, and scope boundaries. If the information in the overlapping area is inconsistent, it is classified according to the conflict type (fact / logic / semantics, etc.). The original text complete context is checked first to determine whether the difference is caused by "time change" or "perspective difference". Information in the overlapping area that is consistent with the global theme is retained. The same content in the overlapping area is merged. The unique information of each block's non-overlapping area is retained to form the corrected block result.

[0058] The correction results of adjacent blocks are associated by topic to ensure logical coherence. Based on block metadata (such as core topics and hierarchical relationships), the local results are linked into a complete chain, and the relationship between each part is marked (such as "budget adjustment affected the execution schedule"). The integrated results are re-entered into the system, and their corresponding consistency and completeness are analyzed. The specific consistency is represented by whether the corrected information is consistent with the global logic of the original text, and the completeness is represented by whether key details in non-overlapping areas are omitted. At the same time, secondary processing results are generated.

[0059] Example 2

[0060] Please see Figure 2 This application provides a long text processing device based on a large model, including a text classification module, a chunking processing module, a conflict analysis module, a conflict resolution module, an overlap processing module, and a result integration module, combined with the appendix. Figure 2 It can be seen that the information between the above functional modules is transmitted in one direction only;

[0061] Text classification module: Obtain long text information, compare the text length with preset values ​​K1 and K2. If the text length > K2, classify it as extra-large text; if K1 < text length ≤ K2, classify it as normal length text; if the text length ≤ K1, classify it as normal long text.

[0062] The chunking module: For normal long text, it chunks the text into fixed lengths; for text of general length, it uses the Sentence-BERT model to calculate the similarity between chunks and merges them into chunks based on semantic similarity; for very large text, it chunks the text into chapter-section-subsection hierarchical structures according to the text format.

[0063] Conflict Analysis Module: Performs factual conflict, logical conflict, semantic conflict, referential conflict, and scope conflict analysis on block information i (i=1, 2, ..., j, where j is the number of blocks). If a conflict exists, a conflict analysis signal is generated; if no conflict exists, a block processing signal is generated.

[0064] Conflict resolution module: For factual conflicts, spaCy and HanLP are used to extract entity-attribute-value triples. The reconciliation of conflicts is determined based on attribute type (absolute / relative). For reconcilable conflicts, the original source is verified first, or an external authoritative source is introduced for correction. For logical conflicts, Stanford CoreNLP and TextGraphs are used to identify causal / conditional relationships, determine if constraints are missing, and correct them using logical expressions or large-scale model reasoning. For semantic conflicts, the semantic dimension of the conflict is located, and conflicts of the same dimension are corrected by clearly referring to standards. For referential conflicts, the direction is determined according to the principle of proximity, semantic association, and global frequency, and the original text annotations or high-frequency related entities are adopted and annotated accordingly. For scope conflicts, scope elements and boundary descriptions are extracted, the inclusion relationship is verified using a hierarchical database, and the scope hierarchy and exceptions / boundaries are annotated.

[0065] Overlap processing module: performs overlap judgment on the block processing result Gi. If overlap exists, it generates an overlap analysis signal, extracts key elements from the overlapping content, compares the processing results, judges the conflict type, and corrects it in combination with the original text context, retaining the same content and the unique content of the non-overlapping area; if there is no overlap, it generates a result integration signal.

[0066] Result Integration Module: Integrates all block processing results (including correction results) in a progressive manner to obtain the integrated result, and re-enters it to generate secondary processing results.

[0067] Example 3

[0068] This application provides a long text processing device based on a large model, including at least one processor; and

[0069] A memory communicatively connected to at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a long text processing method based on a large model.

[0070] Example 4

[0071] Another embodiment of this application provides a computer storage medium for storing a computer program, which, when executed, implements a long text processing method based on a large model.

[0072] Computer storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0073] The data in the above formulas are all calculated using numerical values, without substituting the units of the parameters. In addition, the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0074] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A long text processing method based on a large model, characterized in that, The method specifically includes the following steps: S1. Obtain long text information, classify it according to the comparison results of the text length with preset values ​​K1 and K2, and divide the long text into blocks for different categories to generate block information. S2. Label the obtained blocks, perform block conflict analysis on the block information, and if there is a block conflict, obtain the corresponding conflict and generate a conflict analysis signal; otherwise, do not process it. S3. Process the generated conflict analysis signals to obtain the corresponding conflicts. Process the conflicts according to their different types, including factual conflicts, logical conflicts, semantic conflicts, referential conflicts, and scope conflicts, to obtain block-based processing results. The specific processing method is as follows: Analyze the cases where the conflict is a factual conflict, locate the core entity and corresponding attribute of the conflict, determine whether it is an irreconcilable absolute conflict based on the attribute type, extract the entity-attribute-value triple, distinguish between reconcilable and irreconcilable conflicts, for reconcilable conflicts, first check the original source of the long text to see if there are description differences due to different contexts, if so, correct it, otherwise introduce external authoritative sources to verify and correct, and obtain factual conflict analysis information. Analyze the conflict as a logical conflict, identify logical connectors and logical subjects from the segmented information, determine the type of logical conflict, identify causal relationships, check whether the segmented information is missing any constraints for the logic to hold, if there are omissions, complete them with logical expressions, if there are no omissions, determine the cause of the conflict through large model reasoning and correct it, and obtain logical conflict analysis information; The semantic conflict is analyzed, and the semantic dimension of the conflict is located from the segmented information. The semantic dimension includes same-dimensional conflict and different-dimensional conflict. If it is same-dimensional, the reference standard is identified and corrected to obtain semantic conflict analysis information. The system analyzes the conflict of references, extracts reference markers and candidate entities from the blocks, and determines the reference based on the principle of proximity, semantic association, and global frequency. For conflicting references, it prioritizes the use of entities that are clearly marked in the original text or have high-frequency associations, and marks the reference object and basis to generate reference conflict analysis information. For cases where the conflict is a scope conflict, analyze the scope elements and boundary descriptions, verify the inclusion relationship using a hierarchical database, and mark the scope hierarchy and boundaries to generate scope conflict analysis information; S4. Obtain and label the block processing results. Perform overlap judgment on the block processing results. If overlap exists, generate an overlap analysis signal and process it to generate a corrected result; otherwise, generate a result integration signal. Integrate all block processing results progressively to obtain the integrated result, and re-input the integrated result to generate a secondary processing result. The specific processing method is as follows: Obtain the block processing results and label them as Gi. At the same time, perform overlap judgment on the block processing results Gi. If the block processing results overlap, generate an overlap analysis signal; otherwise, if there is no overlap, generate a result integration signal. The generated overlap analysis signal is processed as follows: the corresponding overlap result is obtained, key elements are extracted from the overlap content, the processing results of the block on the overlap area are compared, the conflict type is determined, the complete context of the overlap area in the original text is checked, the same content in the overlap content is obtained, and the unique content of the non-overlapping area of ​​the block is retained, and the correction result is generated.

2. The long text processing method based on a large model according to claim 1, characterized in that, In step S1, the text length is classified according to the comparison results with preset values ​​K1 and K2. If the text length > K2, it is classified as extra-large text; if K1 < text length ≤ K2, it is classified as normal length text; if the text length ≤ K1, it is classified as normal long text. For normal long texts, they are divided into blocks of fixed length; for texts of general length, they are divided into blocks through semantic awareness, that is, by using a model to calculate the similarity between sentences / paragraphs and merging semantically similar units; for very large texts, they are divided into blocks according to structure, that is, by cutting them into levels based on text format.

3. The long text processing method based on a large model according to claim 1, characterized in that, The specific method for performing block conflict analysis on the block information is as follows: The obtained blocks are labeled as i, and i = 1, 2, ..., j, where j represents the number of blocks. Then, block conflict analysis is performed on block information i. The block conflict analysis is specifically performed separately from factual conflict, logical conflict, semantic conflict, referential conflict and scope conflict. If block information i has a block conflict, the corresponding conflict is obtained and a conflict analysis signal is generated. Otherwise, if block information i does not have a block conflict, it is not processed.

4. A long text processing apparatus based on a large model, used to execute the long text processing method according to any one of claims 1-3, characterized in that, It includes a text classification module, a block processing module, a conflict analysis module, a conflict resolution module, an overlap processing module, and a result integration module.

5. The long text processing device based on a large model according to claim 4, characterized in that, The text classification module acquires long texts and, by comparing them with preset values ​​K1 and K2, classifies them into extra-large texts, normal-length texts, and normal-length texts. The block processing module divides normal long texts into blocks of fixed length. For texts of average length, Sentence-BERT is used to merge and divide them into chunks based on semantic similarity. Large text files are divided into blocks according to the chapter-section-subsection hierarchy based on formatting. Conflict Analysis Module: Analyzes the segmented information for factual, logical, semantic, referential, and scope conflicts. If there is a conflict, a conflict analysis signal is generated; if there is no conflict, a segmented processing signal is generated. In the conflict resolution module, factual conflicts are extracted using tools to determine triples, and reconciliation is judged and corrected according to attribute type; logical conflicts are identified using tools to complete constraints or correct large model reasoning; semantic conflicts are located in dimensions, and same-dimensional conflicts are corrected according to reference standards; referential conflicts are determined and labeled according to proximity, semantic association, and global frequency. Extract features for scope conflicts, validate them using a hierarchical database, and label the hierarchy and exceptions. The overlap processing module determines whether the block results overlap. If there is overlap, it extracts elements, compares and corrects them, and retains the same content and the unique content of the non-overlapping areas. If there is no overlap, it generates an integration signal. The results integration module progressively integrates the results of the segmented processing, generates the integrated result, and performs secondary input processing.

6. A long text processing electronic device based on a large model, characterized in that, At least one processor; And a memory communicatively connected to at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-3.

7. A long text processing medium based on a large model, characterized in that, Used to store computer programs, which, when executed, are used to implement the method described in any one of claims 1-3.

Images