Document self-adaptive sharding method and computer device

Abstract

This application discloses a document adaptive segmentation method and computer device. The method includes: acquiring standardized text and its corresponding layout feature set and the proportion of structured data; inputting the standardized text into a first large language model, outputting a semantic unit set; inputting the semantic unit set into the first large language model, outputting a semantic relation graph; determining a first set of proposed cutting points based on the semantic unit set and the layout feature set; determining multiple text segments based on the layout feature set, the proportion of structured data, the first set of proposed cutting points, a preset first scoring threshold, the semantic unit set, a segmentation constraint set, and a second large language model; generating structured segmentation data based on the first large language model, the semantic relation graph, and the multiple text segments; generating an internal coherence score for the structured segmentation data, determining that if the internal coherence score is greater than a preset second scoring threshold, and using the structured segmentation data as the optimal segmentation data. This application can improve the accuracy of document segmentation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a document adaptive fragmentation method and computer device. Background Technology

[0002] In the field of computer technology, document fragmentation is a crucial preliminary step for downstream applications such as large-scale text retrieval, knowledge graph construction, and retrieval augmentation generation (RAG). Currently, mainstream document processing technologies mainly rely on rule-based static segmentation methods, the most typical of which are using sliding windows of fixed character length or simple punctuation marks such as line breaks and semicolons as physical segmentation markers. The core logic of these methods is to treat the document as a linear stream of characters, and their segmentation process is completely detached from the business logic and narrative structure inherent in the document content. Due to the lack of perception of the deep semantic boundaries of text in existing technologies, mechanical breaks often occur in the middle of complete logical chains or core factual statements when processing long, highly specialized, or structurally complex heterogeneous documents. This "blind segmentation" phenomenon results in severely fragmented fragmented fragmentation results. The originally coherent context is abruptly separated due to physical truncation, causing the loss of key semantic information, and making individual fragments semantic islands due to the loss of context.

[0003] Furthermore, traditional document fragmentation mechanisms exhibit significant lag and rigidity when dealing with diverse document characteristics and ever-changing business scenarios. Because their fragmentation parameters (such as word count thresholds) are typically preset, fixed empirical values, they cannot dynamically adjust to differences in narrative style, information density, and text / image layout within the document. When faced with complex documents containing large amounts of professional data, nested logic, or non-standard paragraphs, static fragmentation struggles to balance the accuracy of information extraction with the logical integrity of the content, failing to retain sufficient semantic depth while meeting storage constraints. This lack of flexibility results in extremely poor universality of document fragmentation across different industry applications, generally low accuracy of fragmentation results, and a lack of necessary logical connections between fragments, severely restricting the performance and data utilization efficiency of subsequent intelligent processing stages. Summary of the Invention

[0004] The purpose of this application is to provide a document adaptive fragmentation method and computer device to solve the problem of low document fragmentation accuracy in the prior art.

[0005] To achieve the above objectives, the first aspect of this application provides a document adaptive fragmentation method, the method comprising: Obtain standardized text, its corresponding set of typesetting features, and the proportion of structured data; The standardized text is input into the first language model, and the output is a set of semantic units. Input the set of semantic units into the first language model and output a semantic relation graph; The first set of proposed cutting points is determined based on the set of semantic units and the set of typesetting features; Based on the typesetting feature set, the proportion of structured data, the first set of proposed cutting points, the preset first scoring threshold, the semantic unit set, the fragmentation constraint set, and the second major language model, the standardized text is segmented to obtain multiple text fragments; Structured fragmented data is generated based on the first major language model, semantic relation graph, and multiple text fragments; The internal coherence score of the structured fragmented data is generated by the first language model. If the internal coherence score is greater than the preset second score threshold, the structured fragmented data is selected as the optimal fragmented data.

[0006] In this embodiment, the steps for generating a fragmentation constraint set include: obtaining the downstream business task type; determining the corresponding business base coefficient, task complexity coefficient, and task length coefficient based on the downstream business task type and a preset mapping table; the preset mapping table stores the mapping relationship between the business task type and the business base coefficient, task complexity coefficient, and task length coefficient; using the total number of semantic units in the semantic unit set as the total number of semantic units; determining the minimum number of semantic units based on the total number of semantic units, the business base coefficient, and the task complexity coefficient; determining the maximum character length based on the proportion of structured data, the task length coefficient, and the preset base character length; and generating a fragmentation constraint set based on the minimum number of semantic units, the maximum character length, and the semantic unit combination constraints.

[0007] In this embodiment, the step of segmenting standardized text into multiple text segments based on a typesetting feature set, the proportion of structured data, a first set of proposed cutting points, a preset first scoring threshold, a set of semantic units, a set of segmentation constraints, and a second major language model includes: determining the comprehensive semantic coherence score of each first proposed cutting point in the standardized text based on the typesetting feature set, the proportion of structured data, the standardized text, the set of semantic units, the first set of proposed cutting points, and the second major language model; determining a second set of proposed cutting points based on the comprehensive semantic coherence score of each first proposed cutting point in the standardized text and the preset first scoring threshold; and repeatedly executing the following steps, using each second proposed cutting point in the second set of proposed cutting points as the current cutting point. The steps are as follows, until all the second proposed cutting points in the second proposed cutting point set have been traversed: The standardized text is cut according to the current cutting point and the previous cutting point to obtain the candidate text interval between them as preliminary text fragments; the semantic unit subset corresponding to the preliminary text fragments in the semantic unit set is determined; the semantic unit subset satisfies the fragmentation constraint set, and the current cutting point is taken as the valid cutting point; if the semantic unit subset satisfies the fragmentation constraint set for the first time, the starting point of the standardized text is determined as the previous cutting point; the current cutting point is updated to the previous cutting point, and the next second proposed cutting point of the current cutting point is updated to the new current cutting point; the standardized text is cut according to the valid cutting point and the starting point to obtain multiple text fragments.

[0008] In this embodiment of the application, based on the typesetting feature set, the proportion of structured data, standardized text, semantic unit set, first set of proposed cutting points, and second language model, the comprehensive semantic coherence score of each proposed cutting point in the standardized text is determined by: determining the comprehensive semantic coherence score according to the following formula:

[0009] in, A comprehensive score for semantic coherence; The first text segment in the standardized text that ends at the first proposed cutting point; The second text segment in the standardized text, starting from the first proposed cutting point; The semantic coherence score is given between the first and second text segments output by the second largest language model. The first type label is the semantic unit closest to the first intended cutting point within the first semantic unit subset corresponding to the first text segment in the semantic unit set. The second type label is the semantic unit closest to the first intended cutting point within the second semantic unit subset corresponding to the second text fragment in the semantic unit set. The business weight corresponding to the combined type label generated based on the first type label and the second type label in the preset weight allocation table; These are the fusion weight coefficients obtained based on the proportion of structured data; The typesetting feature matching score is obtained based on the position of the first proposed cutting point in the standardized text and the typesetting feature set.

[0010] In this embodiment of the application, the semantic unit combination constraint satisfies the following formula:

[0011] in, Constraints for semantic unit composition; For the first semantic unit in the subset One semantic unit; For the first The business weight corresponding to the type label of each semantic unit in the preset weight allocation table; The preset weight threshold; This represents the total number of semantic units in the semantic unit subset.

[0012] In this embodiment of the application, determining the minimum number of semantic units based on the total number of semantic units, the business basis coefficient, and the task complexity coefficient includes: determining the minimum number of semantic units according to the following formula:

[0013] in, The minimum number of semantic units; For business basis coefficients; This represents the total number of semantic units. This represents the task complexity coefficient.

[0014] In this embodiment, determining the maximum character length based on the proportion of structured data, the task length coefficient, and the preset basic character length includes: determining the maximum character length according to the following formula:

[0015] in, Maximum character length; Preset base character length; The proportion of structured data; This is the task length coefficient.

[0016] In this embodiment of the application, the step of generating structured fragmented data based on the first large language model, the semantic relationship graph, and multiple text fragments includes: generating semantic tags and content summaries for each text fragment using the first large language model; establishing mapping relationships between text fragments based on the semantic relationship graph to obtain fragment association pointers between text fragments; and generating structured fragmented data based on multiple text fragments, the semantic tags and content summaries of each text fragment, and the fragment association pointers.

[0017] In this embodiment of the application, the step of inputting standardized text into the first large language model and outputting a set of semantic units includes: inputting standardized text into the first large language model and outputting a context-aware representation vector; guiding the first large language model to detect the semantic boundaries of the standardized text based on the context-aware representation vector through prompting engineering, and outputting a set of semantic units.

[0018] A second aspect of this application provides a computer device, comprising: The memory is configured to store instructions; and The processor is configured to retrieve instructions from memory and to implement the methods described above when executing instructions.

[0019] Through the aforementioned technical solution, by multi-dimensionally perceiving standardized text, layout features, and the proportion of structured elements, the first major language model is used to deeply probe semantic boundaries and construct a global semantic relationship graph. This transforms traditional mechanical character stream segmentation into precise recognition based on logically autonomous units, thereby avoiding logical fragmentation caused by ignoring semantic coherence at the source. Furthermore, a dual verification is achieved through semantic coherence evaluation by the second major language model and dynamically generated segmentation constraint sets. This allows the segmentation action to adaptively adjust according to business tasks, document features, and logical tightness, ensuring that candidate segmentation positions are always anchored at the intersection of semantics and physical structure. Based on this, structured segmented data carrying semantic tags, content summaries, and association pointers is generated, and an internal coherence scoring mechanism based on the first major language model is introduced for closed-loop final review. This ensures that the final optimal segmented data, while complying with physical scale requirements, possesses extremely high logical self-consistency and contextual fidelity, thus significantly improving the accuracy of text segmentation.

[0020] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1A flowchart illustrating an adaptive document fragmentation method according to an embodiment of this application is shown schematically. Figure 2 The schematic diagram illustrates a structural diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0023] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0024] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0025] The acquisition, transmission, storage, use, and processing of data in this application comply with relevant laws and regulations. Furthermore, it should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0026] It should be noted that all data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are information and data authorized by the client or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0027] Figure 1 A flowchart illustrating an embodiment of a document adaptive fragmentation method according to this application is shown schematically. Figure 1 As shown in the figure, this application provides a document adaptive fragmentation method, which may include the following steps.

[0028] Step 101: Obtain standardized text, corresponding typesetting feature sets, and the proportion of structured data.

[0029] In this embodiment, standardized text refers to a data stream to be segmented that has undergone preliminary cleaning and format alignment and possesses a unified encoding standard. The layout feature set refers to the physical layout parameters and hierarchical style identifiers extracted from the document, encompassing character spacing, paragraph indentation, heading levels, and various visual segmentation markers. The proportion of structured data is a quantitative statistical indicator reflecting the distribution ratio between strongly rule-based content such as tables and numerical lists and conventional narrative descriptions in the document.

[0030] Step 102: Input the standardized text into the first language model and output a set of semantic units.

[0031] In this embodiment, the first language model, as a generative artificial intelligence architecture with deep semantic parsing and knowledge detection capabilities, is responsible for identifying the logical structure within the text. The semantic unit set refers to the sequence of the smallest semantic information carriers detected by the model based on semantic coherence, which are logically independent and cannot be further divided.

[0032] By performing deep semantic boundary detection on standardized text and constructing a set of semantic units, a judgment basis based on logical connotation rather than physical length is established for the subsequent segmentation process, which fundamentally solves the problems of semantic fragmentation and contextual discontinuity caused by traditional mechanical segmentation.

[0033] Step 103: Input the set of semantic units into the first large language model and output the semantic relationship graph.

[0034] In this embodiment, the semantic relationship graph refers to a knowledge grid that structurally represents the logical connections, causal relationships, and narrative order among the independent semantic expression carriers within a document. Through the automated parsing of the deep meaning of the context using this model, discrete unit information is transformed into a logical topological structure with clear associative pointers, providing a basis for subsequent identification of association pointers between segments.

[0035] Step 104: Determine the first set of cut points based on the set of semantic units and the set of typographic features.

[0036] In this embodiment, the first set of proposed cutting points refers to the set of candidate segmentation position coordinates determined jointly by semantic logical boundaries and physical layout identifiers. Specifically, the system spatially maps and aligns the logical intersections of each unit in the semantic unit set with physical segmentation markers such as line breaks, paragraph breaks, or page breaks in the layout feature set, thereby locking in a set of physical position sequences that initially have the potential for segmentation in the standardized text.

[0037] By constructing a semantic relationship graph, the logical dependencies between information nodes are clarified. Combined with the first set of proposed cutting points determined by physical layout features, it is ensured that the candidate cutting positions are always anchored on nodes that comply with both semantic logic and document structure. This avoids semantic fragmentation and contextual logical discontinuity caused by traditional blind cutting from the source.

[0038] Step 105: Based on the typesetting feature set, the proportion of structured data, the first set of proposed cutting points, the preset first scoring threshold, the semantic unit set, the fragmentation constraint set, and the second major language model, the standardized text is segmented to obtain multiple text fragments.

[0039] In this embodiment, the preset first scoring threshold refers to a pre-defined numerical boundary used to measure the quality of segmentation points, serving as a definite threshold for selecting segmentation positions with semantic coherence. The segmentation constraint set refers to a multi-dimensional set of quantitative criteria constructed based on downstream application scenarios and document statistical characteristics. It covers judgment indicators for segmentation size, information value, and character length to ensure the compliance of the output content. The second major language model refers to a computational architecture with deep context awareness capabilities, responsible for providing a quantitative evaluation of the semantic smoothness of candidate segmentation positions through complex logical reasoning. Text segmentation refers to the final output data block formed after standardized text undergoes multiple verifications and physical separation, possessing logical autonomy and meeting various preset standards.

[0040] The system utilizes a second language model to calculate the semantic coherence of each candidate point in the first set of proposed cut points. Taking into account the objective influence of the layout feature set and the proportion of structured data, it determines the semantic coherence of each candidate point and selects the optimal boundary based on the comparison between the obtained score and a preset first scoring threshold. By physically dividing the standardized text into intervals, it identifies the semantic unit subsets corresponding to each candidate text interval and matches them one by one with the quantitative indicators in the fragmentation constraint set. Only when the semantic unit combination and physical attributes within the current text range fully meet the criteria are the corresponding positions confirmed as valid cut points, thus outputting multiple text fragments.

[0041] By introducing a semantic coherence scoring mechanism based on a deep learning architecture, and using dynamically generated quantitative constraint sets to verify the compliance of each pre-shard, the splitting action not only meets the physical format requirements, but also accurately anchors at the natural termination point of the business logic, thereby effectively avoiding information fragmentation and semantic loss.

[0042] Step 106: Generate structured fragmented data based on the first major language model, semantic relation graph, and multiple text fragments.

[0043] In this embodiment, structured fragmented data refers to a comprehensive information representation entity with a standard format and logical association features, formed by deeply integrating originally discrete physical text blocks with their corresponding multi-dimensional business attributes and logical mapping information. It not only covers the original content of multiple text fragments themselves, but also encapsulates semantic tags extracted by the first major language model for parsing each fragment's content to define the business scope, as well as content summaries for extracting core intent. Simultaneously, this data entity also includes mapping relationships established between text fragments based on a pre-sequence semantic relationship graph, and logically directional fragment association pointers derived from this, thereby reconstructing isolated text fragments into digital knowledge units with contextual awareness. Through the first major language model, which possesses high generalization and semantic inference capabilities, automated feature annotation and thematic compression are performed on each text fragment. By referencing the pre-set logical framework in the semantic relationship graph, causal, progressive, or supplementary association paths are reconstructed between physically separated fragments, forming precise fragment association pointers. Ultimately, the system integrates and encapsulates text fragmentation, semantic tags, content summaries, and fragment association pointers to produce standardized structured fragmented data.

[0044] By attaching rich semantic metadata to each shard and establishing a global logical mapping pointer, it is ensured that when a shard is called by a downstream system, its narrative context and business intent in the original document can be restored in real time, which completely solves the technical pain points of information silos and severe fragmentation of context under traditional sharding methods.

[0045] Step 107: Generate an internal coherence score for the structured fragmented data using the first major language model. If the internal coherence score is greater than a preset second score threshold, the structured fragmented data is selected as the optimal fragmented data.

[0046] In this embodiment, the internal coherence score refers to a closed-loop quality evaluation metric for encapsulated structured fragmented data, using a first-class language model with advanced logic auditing and semantic evaluation capabilities. This metric focuses on evaluating the fit between text content and its corresponding semantic tags and content summaries, as well as the smoothness of the contextual logic chain formed by fragment association pointers, thereby determining the logical consistency of the fragmentation scheme from a macroscopic perspective. The preset second scoring threshold is a pre-set final quality limit used to measure whether the fragmentation results meet the delivery standards; it serves as the core control gate for the system's semantic integrity. Optimal fragmented data refers to structured information entities that have undergone final semantic verification and whose logical indicators all exceed the predetermined threshold, representing the highest accuracy output of the fragmentation process under the current constraints. The system calls the first-class language model to perform end-to-end semantic review of the generated structured fragmented data. Based on the semantic relationship graph and metadata mapping relationships between fragments, the model calculates a quantified internal coherence score. Subsequently, by automatically comparing this score with the preset second scoring threshold, the target fragmentation scheme with rigorous logic and accurate semantic alignment is identified. Once the score is determined to meet the quality threshold requirements, the system will remove the unqualified fragments and complete the final locking and output confirmation of the optimal fragment data.

[0047] By introducing a secondary quality control feedback mechanism based on the primary language model, internal coherence scoring is used to logically correct and semantically verify the initially generated segmentation results. Combined with dynamic filtering using a high-standard preset second scoring threshold, this ensures that the optimal output segmentation data accurately reproduces the business logic of the original document. Technically, this eliminates segmentation errors caused by semantic fragmentation or loss of background information, thereby significantly improving the accuracy and reliability of the final output in complex application scenarios.

[0048] Through the aforementioned technical solution, by multi-dimensionally perceiving standardized text, layout features, and the proportion of structured elements, the first major language model is used to deeply probe semantic boundaries and construct a global semantic relationship graph. This transforms traditional mechanical character stream segmentation into precise recognition based on logically autonomous units, thereby avoiding logical fragmentation caused by ignoring semantic coherence at the source. Furthermore, a dual verification is achieved through semantic coherence evaluation by the second major language model and dynamically generated segmentation constraint sets. This allows the segmentation action to adaptively adjust according to business tasks, document features, and logical tightness, ensuring that candidate segmentation positions are always anchored at the intersection of semantics and physical structure. Based on this, structured segmented data carrying semantic tags, content summaries, and association pointers is generated, and an internal coherence scoring mechanism based on the first major language model is introduced for closed-loop final review. This ensures that the final optimal segmented data, while complying with physical scale requirements, possesses extremely high logical self-consistency and contextual fidelity, thus significantly improving the accuracy of text segmentation.

[0049] In one embodiment, the construction process of the first major language model can be as follows: 1. The first major language model employs a pre-trained Transformer architecture with an extremely large number of parameters. Unlike the second major language model, which focuses on local evaluation, the first major language model is defined as an all-around logic engine, designed to perform complex tasks ranging from micro-boundary detection to macro-relationship modeling. Deep Feature Space: Through pre-training on massive amounts of multimodal technical documents, research papers, and industry standards, the model establishes a semantic representation space that covers a wide range of engineering fields, enabling accurate identification of technical terms and implicit logical levels in standardized text. Reasoning Enhancement Training: During the fine-tuning phase, Chain of Thought training is specifically introduced, enabling it to not only provide results when outputting semantic unit sets or performing logical checks, but also to perform internal reasoning based on the textual context, ensuring that the produced content summaries and semantic tags possess deep meaning.

[0050] 2. The first major language model underwent fine-tuning for the following specific tasks: Semantic boundary detection task: trained using corpora containing a large number of "semantic breakpoint" annotations. Through prompting engineering, the model was guided not only to focus on physical punctuation marks but also to combine context-aware representation vectors to identify semantic intent transition points, thereby outputting a set of logically autonomous semantic units. Knowledge graph construction task: trained the model to learn how to identify causal, transitional, and progressive logical relationships between discrete text blocks and transform them into structured topological descriptions, i.e., generating a semantic relationship graph. Multidimensional attribute extraction task: for the generation of structured fragmented data, the model underwent joint training for information extraction and summary generation, ensuring that the model can automatically attach accurate semantic tags and highly compressed content summaries to each text fragment.

[0051] 3. Consistency Learning: Train the model to recognize the matching degree between "text content" and "metadata". For example, if a shard's label is "fault handling" but the content is "equipment installation", the model needs to be able to identify this semantic conflict. Threshold Calibration Training: By manually scoring and ranking a large number of shard results, guide the model to learn how to produce internally consistent scores that align with business intuition. This score should sensitively reflect whether there are logical gaps in the shards. Only when the score is greater than a preset second scoring threshold is the shard considered the optimal shard data.

[0052] In one embodiment, the construction process of the second major language model can be as follows: 1. To enable the second language model to accurately detect semantic boundaries, a training dataset is first constructed, containing a large number of unstructured and semi-structured documents (such as petrochemical equipment manuals with tables and parameter sets). Positive sample construction: Continuous text fragment pairs within natural paragraphs are extracted from a high-quality corpus and labeled as "coherent." Negative sample construction: Manual or automatic truncation is performed at boundaries between different chapters, different business logics, or where there are significant formatting differences, constructing "logical discontinuity" fragment pairs. Feature injection: Corresponding formatting feature sets (such as HTML tags and Markdown formatting symbols) and structured data proportion identifiers are simultaneously encapsulated in the samples, enabling the model to establish a mapping relationship between physical formatting and semantic logic during the training phase.

[0053] 2. Based on the basic large model, supervised fine-tuning (SFT) is used for special tuning, focusing on optimizing its discrimination logic for segment boundaries: Input construction: The first and second text segments before and after the point to be evaluated in the standardized text are used as input pairs, and the typographical feature description of the point is used as supplementary context. Training objective design: The model not only performs discrimination and classification tasks, but also regression tasks. Through training, it can output a value between [0,1] based on the context-aware representation vector, which is the semantic coherence comprehensive score. This score is used to measure the degree of connection between the first and second text segments in business logic. Loss function optimization: A penalty term for logical breaks is introduced to ensure that the model can give a very low score when facing "logical islands" caused by physical length truncation, and a high score when facing boundaries with complete business meaning.

[0054] 3. Constraint-Aware Training: During training, "task complexity coefficient" and "task length coefficient" are introduced as conditional inputs to guide the model in learning the dynamic fluctuation patterns of semantic coherence scores under different business constraints. Inference Alignment: Prompt engineering optimizes the model's inference path, enabling it to automatically reference the proportion of structured data when generating scores. For example, when a high proportion of structured data is detected, the model automatically increases the weight given to table integrity, ensuring that the output score directly serves the selection of the second set of potential cut points.

[0055] In another embodiment, the training process for the second major language model can be as follows: The formula for calculating the overall score of semantic coherence is as follows: =Semantic logic score + typography matching score. Semantic logic score = Layout matching score = Formula explanation: Based on the comprehensive business semantic logical relevance and the matching degree of layout features, the output is a boundary score of 0-10. A score of Score(b) ≥ 6 is a valid boundary; otherwise, it is adjusted.

[0056] 1,000 soil / groundwater monitoring reports from a company between 2018 and 2022, texts of 5 environmental standards including GB36600-2018, and a terminology database for the field of environmental monitoring (containing 3,000+ professional terms).

[0057] Training methods: Supervised fine-tuning + Prompt engineering.

[0058] Loss function: ,in The loss is used for semantic unit classification. This includes losses related to environmental business logic (such as the logical matching loss of "detection method → ​​monitoring result").

[0059] Output: Text before and after the given boundary b ( For the last sentence of segment A, (For the first sentence of segment B), the second language model outputs a basic semantic coherence score of 0-10.

[0060] like When ∈{(S2,S3),(S3,S4),(S4,S6)}, =0.9.

[0061] like When ∈{(S1,S2),(S3,S5),(S5,S6)}, =0.7.

[0062] If other semantic units are combined =0.5.

[0063] Logical basis: In the environmental monitoring report, the core business logic chain is "detection method → ​​monitoring results", "monitoring results → exceeding standards data", and "exceeding standards data → conclusions and recommendations", which has the highest weight; "monitoring indicators → detection methods", etc. are auxiliary logic chains with the next highest weight.

[0064]

[0065] Chapter title separation feature (M=1 if b is below the chapter title, otherwise 0), weight w1=0.4; : End feature of table / chart (M=1 if b is an empty row below the table / chart, otherwise 0), weight w2=0.35; Paragraph indentation variation characteristics (M=1 if the difference in indentation between paragraphs before and after b is ≥2 characters, otherwise 0), weight w3=0.25; Output: 0-1 points for layout matching (the higher the score, the better the boundary matches the report layout logic).

[0066] Taking the first proposed cutting point b = "below the summary information table of soil detection indicators in the refinery area in Table 8-3" as an example: Text before and after the boundary: (Last sentence of segment A): "A total of 22 indicators were detected in 41 soil samples from the refinery area, among which the arsenic content at point ZS05 was 134 mg / kg" (Semantic unit S3 = monitoring results). (First sentence of segment B): "According to the screening values ​​of Class II land use in GB36600-2018, the standard limit for arsenic is 60 mg / kg, and the exceedance at this location is 1.23 times" (Semantic unit S4 = Exceedance data). With structured data accounting for D=0.6, the calculated λ=0.3+0.2×0.6=0.42; Layout features: b is the blank line below the table, M(b,P1)=0, M(b,P2)=1, M(b,P3)=0, StructMatch(b)=0.4×0+0.35×1+0.25×0=0.35.

[0067] The second major language model outputs a basic score: Input The model identified the core business logic of "monitoring results → data exceeding the standard" and output a base score of 8.5. Business weight =0.9; Semantic logic score = 8.5 × 0.9 = 7.65 points; The layout matching score is approximately 0.147 points (0.42 × 0.35). comprehensive =7.65+0.147≈7.797 points≥6 points, the first proposed cutting point is valid, and the first proposed cutting point is retained.

[0068] If the first proposed cutting point b = "arsenic content at point ZS05 is 134 mg / kg": “The arsenic content at site ZS05 is 134” (semantic break); "mg / kg, according to the screening value for Class II land use in GB36600-2018"; The second largest language model outputs a base score of 3.2 (incomplete semantics, logical breaks). Semantic logic score = 3.2 × 0.9 = 2.88 points; comprehensive =2.88 + 0.42 × 0.1 (low layout matching) ≈ 2.92 points < 6 points; Adjustment strategy: Skip the current first proposed cut point, determine the semantic coherence comprehensive score at the next first proposed cut point (i.e., "the score at this point exceeds the limit by 1.23 times"), and then recalculate. =7.2 points ≥ 6 points, retain the first proposed cutting point. All retained first proposed cutting points are used as the second proposed cutting point set.

[0069] In this embodiment, the steps for generating a fragmentation constraint set include: obtaining the downstream business task type; determining the corresponding business base coefficient, task complexity coefficient, and task length coefficient based on the downstream business task type and a preset mapping table; the preset mapping table stores the mapping relationship between the business task type and the business base coefficient, task complexity coefficient, and task length coefficient; using the total number of semantic units in the semantic unit set as the total number of semantic units; determining the minimum number of semantic units based on the total number of semantic units, the business base coefficient, and the task complexity coefficient; determining the maximum character length based on the proportion of structured data, the task length coefficient, and the preset base character length; and generating a fragmentation constraint set based on the minimum number of semantic units, the maximum character length, and the semantic unit combination constraints.

[0070] In this embodiment, the downstream business task type refers to a specific business scenario where in-depth processing or analysis of text data is subsequently performed, which determines the resolution and information carrying requirements of the sharding. Based on the downstream business task type and a preset mapping table, corresponding business base coefficients, task complexity coefficients, and task length coefficients are determined. The preset mapping table is a configuration information entity that stores the correspondence between different business requirements and specific quantitative control parameters, used to transform abstract application tasks into concrete computational benchmarks. The total number of semantic units in the semantic unit set is taken as the total number of semantic units, reflecting the information scale and distribution density of the text to be processed in the global logical dimension. The minimum number of semantic units is determined based on the total number of semantic units, the business base coefficient, and the task complexity coefficient. The minimum number of semantic units refers to the lower limit of logical unit carrying capacity set to ensure that the produced text entities have independent business meaning. The maximum character length is determined based on the proportion of structured data, the task length coefficient, and the preset basic character length. The maximum character length refers to the upper limit of the number of physical characters allowed to be carried by a single sharding result, provided that the load performance of the downstream system is met. A fragmentation constraint set is generated based on the minimum number of semantic units, the maximum character length, and semantic unit combination constraints. The fragmentation constraint set refers to a set of multi-dimensional quantitative criteria constructed by physical size constraints and logical value indicators to determine the compliance of the fragmentation results.

[0071] By identifying downstream business task types and retrieving preset mapping tables, adaptive alignment between the sharding standard and actual application scenarios is achieved. The minimum number of semantic units is dynamically adjusted based on the total number of semantic units, ensuring that each segmented target shard has sufficient semantic depth, thus preventing logical fragmentation caused by overly fragmented shards. Simultaneously, the maximum character length is flexibly adjusted based on the proportion of structured data, allowing the physical sharding boundary to adaptively match the density of document content layout. This dynamically generated sharding constraint set based on task characteristics and document statistical attributes provides multi-dimensional quantitative compliance guidelines for subsequent precise sharding, effectively improving the accuracy and reliability of the sharding output.

[0072] In one embodiment, when the system processes a petrochemical pump unit operation monitoring report, it first identifies the corresponding downstream business task type as automatic fault root cause diagnosis. The system automatically matches and determines the corresponding business base coefficient by searching a preset mapping table. The task complexity coefficient is 6. The value is 1.2 and the task length coefficient. It is 1.3.

[0073] The system performs a full-domain scan of the input data to calculate the total number of semantic units in the set of semantic units corresponding to the current standardized text. There are 450 items, and the proportion of structured data such as tables and parameter sets included in the text is calculated based on the layout characteristics. It is 0.15.

[0074] Based on the aforementioned real-time acquired dynamic parameters, the system utilizes The minimum number of semantic units was calculated. The maximum character length is calculated as 8; simultaneously, based on the preset base character length of 500 characters, the maximum character length is determined. It contains 728 characters.

[0075] Based on this, the system further assigns corresponding business weights to different types of tags, such as "vibration frequency", "bearing temperature" and "diagnostic conclusion". And set a preset weight threshold. It is 0.9.

[0076] Finally, the system integrates the calculated minimum number of semantic units and maximum character length with the semantic unit combination constraints composed of business weights to generate a fragmentation constraint set for the current document. This constraint set is used to perform multi-dimensional verification on each pre-processed text fragment during the segmentation process, ensuring that the generated text fragments meet physical storage limitations while possessing sufficient business value and logical depth, thereby significantly improving the accuracy of text fragmentation.

[0077] In this embodiment, the step of segmenting standardized text into multiple text segments based on a typesetting feature set, the proportion of structured data, a first set of proposed cutting points, a preset first scoring threshold, a set of semantic units, a set of segmentation constraints, and a second major language model includes: determining the comprehensive semantic coherence score of each first proposed cutting point in the standardized text based on the typesetting feature set, the proportion of structured data, the standardized text, the set of semantic units, the first set of proposed cutting points, and the second major language model; determining a second set of proposed cutting points based on the comprehensive semantic coherence score of each first proposed cutting point in the standardized text and the preset first scoring threshold; and repeatedly executing the following steps, using each second proposed cutting point in the second set of proposed cutting points as the current cutting point. The steps are as follows, until all the second proposed cutting points in the second proposed cutting point set have been traversed: The standardized text is cut according to the current cutting point and the previous cutting point to obtain the candidate text interval between them as preliminary text fragments; the semantic unit subset corresponding to the preliminary text fragments in the semantic unit set is determined; the semantic unit subset satisfies the fragmentation constraint set, and the current cutting point is taken as the valid cutting point; if the semantic unit subset satisfies the fragmentation constraint set for the first time, the starting point of the standardized text is determined as the previous cutting point; the current cutting point is updated to the previous cutting point, and the next second proposed cutting point of the current cutting point is updated to the new current cutting point; the standardized text is cut according to the valid cutting point and the starting point to obtain multiple text fragments.

[0078] In this embodiment, the semantic coherence comprehensive score refers to a quantitative evaluation value characterizing the quality of logical connection and the smoothness of context transition at a specific boundary position, which is obtained by a computing architecture with deep logical reasoning capabilities based on the text context. The second set of proposed cutting points refers to the preferred set of boundary coordinates with high cutting rationality retained after semantic quality screening among the initially determined candidate sites. The current cutting point refers to a specific physical site in the logical verification process during the traversal of the candidate boundary set. The previous cutting point refers to a reference position in the cutting sequence that is located before the current site and has been determined as a logical starting point or valid boundary. The candidate text interval refers to a physical character fragment enclosed by two specific boundary coordinates within the standardized text. The preliminary text fragment refers to a candidate data entity composed of specific text intervals, which serves as the constraint verification object and carries the original information to be verified. The semantic unit subset refers to a combination of specific semantic information carriers identified from the global set whose physical location falls within the scope of the preliminary text fragment. The valid cutting point refers to a legally valid fragment boundary that has passed multi-dimensional compliance verification and is ultimately confirmed to meet business logic and scale requirements. The starting point refers to the physical starting coordinates of the current segment to be processed in the standardized text within the document segmentation sequence.

[0079] By implementing refined boundary screening based on semantic scoring and dynamically verifying the semantic unit subsets contained in each prepared text segment using a multi-dimensional set of constraints, the process ensures that the segmentation always occurs at logically autonomous and semantically complete boundaries. This refined processing flow, which determines the compliance of global segmentation based on the state of local semantic subsets, fundamentally eliminates semantic biases caused by mechanical segmentation and effectively avoids missegmentation or omissions caused by ignoring deep content relationships, thereby significantly improving the accuracy of segmentation output.

[0080] In one embodiment, taking the processing of a standardized text involving petrochemical equipment fault diagnosis as an example, the system performs a refined cutting process on the identified first set of potential cutting points. First, the system targets a specific first potential cutting point in the first set of potential cutting points. To evaluate, the second largest language model is used to retrieve the first text fragments before and after it. With the second text fragment And identify the corresponding first type of label. With the second type of label The semantic coherence score is calculated using the following formula:

[0081] pass Determine the typesetting feature matching score; in this formula, This represents the total number of layout features with significant indicative meaning selected from the layout feature set. To improve the accuracy of segmentation, the system does not use all fragmented information from the layout feature set for calculation. Developers or the system's default logic will select features from the layout feature set that have significant indicative meaning for "semantic breakpoints" as... . The feature weight score corresponding to the j-th type of typesetting feature; This is a Boolean value. It is set to 1 if the physical location of the first proposed cutting point b matches a corresponding typesetting feature, and 0 otherwise. The system evaluates the physical segmentation suitability of the first proposed cutting point b at the physical layout level by quantifying the physical segmentation strength (such as paragraph breaks, page breaks, specific punctuation marks, etc.) within the typesetting feature set. If the semantic coherence score of this point is 0.95, exceeding the preset first score threshold, the system identifies it and adds it to the second set of proposed cutting points.

[0082] Subsequently, the system enters a loop traversal phase, sequentially using each of the second proposed cutting points in the second set of proposed cutting points as the current cutting point. For a given current cutting point, the system segments the standardized text based on its physical position relative to the previous cutting point (initially the document starting point), obtaining the candidate text interval between the two as preliminary text fragments. Simultaneously, the system determines the semantic unit subset corresponding to this preliminary text fragment in the semantic unit set and verifies whether it satisfies the fragmentation constraint set. The verification process incorporates dynamically calculated quantitative indicators: verifying the total number of semantic units within the subset. Has the formula been passed? Calculated minimum number of semantic units (e.g., N=4); check if the character length of this interval does not exceed the limit specified by the formula. Determined maximum character length (e.g., L=417); and extract the business weights associated with each unit in the subset and sum them up to determine whether the semantic unit combination constraint is satisfied.

[0083] Once a subset of semantic units is determined to satisfy the fragmentation constraint set, the system records the current cut point as a valid cut point. Since this subset is the first compliant logical block, the system determines the starting point of the standardized text as the previous cut point, then updates the current cut point to the previous cut point, and automatically updates the next second proposed cut point to the new current cut point to continue traversal. After completing all traversals, the system physically segments the standardized text based on the recorded valid cut points and starting points, ultimately outputting multiple text fragments with rigorous semantic logic and significantly improved accuracy. This embodiment effectively solves the semantic fragmentation problem caused by traditional mechanical segmentation through deep coupling of multi-dimensional semantic scoring and dynamic constraint verification, significantly improving the accuracy of text fragmentation.

[0084] In this embodiment of the application, based on the typesetting feature set, the proportion of structured data, standardized text, semantic unit set, first set of proposed cutting points, and second language model, the comprehensive semantic coherence score of each proposed cutting point in the standardized text is determined by: determining the comprehensive semantic coherence score according to the following formula:

[0085] in, A comprehensive score for semantic coherence; The first text segment in the standardized text that ends at the first proposed cutting point; The second text segment in the standardized text, starting from the first proposed cutting point; The semantic coherence score is given between the first and second text segments output by the second largest language model. The first type label is the semantic unit closest to the first intended cutting point within the first semantic unit subset corresponding to the first text segment in the semantic unit set. The second type label is the semantic unit closest to the first intended cutting point within the second semantic unit subset corresponding to the second text fragment in the semantic unit set. The business weight corresponding to the combined type label generated based on the first type label and the second type label in the preset weight allocation table; These are the fusion weight coefficients obtained based on the proportion of structured data; The typesetting feature matching score is obtained based on the position of the first proposed cutting point in the standardized text and the typesetting feature set.

[0086] In the embodiments of this application, .in, The percentage of structured data (0≤D≤1) is determined by the following logic: the higher the percentage of structured data, the greater the reference value of the layout features for the boundaries.

[0087] In this embodiment of the application, the semantic unit combination constraint satisfies the following formula:

[0088] in, Constraints for semantic unit composition; For the first semantic unit in the subset One semantic unit; For the first The business weight corresponding to the type label of each semantic unit in the preset weight allocation table; The preset weight threshold; This represents the total number of semantic units in the semantic unit subset.

[0089] In this embodiment of the application, determining the minimum number of semantic units based on the total number of semantic units, the business basis coefficient, and the task complexity coefficient includes: determining the minimum number of semantic units according to the following formula:

[0090] in, The minimum number of semantic units; For business basis coefficients; This represents the total number of semantic units. This represents the task complexity coefficient.

[0091] In this embodiment, determining the maximum character length based on the proportion of structured data, the task length coefficient, and the preset basic character length includes: determining the maximum character length according to the following formula:

[0092] in, Maximum character length; Preset base character length; The proportion of structured data; This is the task length coefficient.

[0093] In this embodiment of the application, the step of generating structured fragmented data based on the first large language model, the semantic relationship graph, and multiple text fragments includes: generating semantic tags and content summaries for each text fragment using the first large language model; establishing mapping relationships between text fragments based on the semantic relationship graph to obtain fragment association pointers between text fragments; and generating structured fragmented data based on multiple text fragments, the semantic tags and content summaries of each text fragment, and the fragment association pointers.

[0094] In this embodiment, semantic tags refer to classification identifiers automatically extracted based on the content connotation of each text segment, used to define its business scope or theme affiliation. Content summaries are concise descriptions generated after refining the main idea information of a text segment using a computational architecture with high summarization capabilities; they serve as condensed information reflecting the key facts of the segment. Mapping relationships are established between text segments based on a semantic relationship graph, resulting in segment association pointers between text segments. Segment association pointers are logically indicative associations established between physically discrete text blocks based on the logical context of the entire document, representing causality, sequence, or supplementation; they are responsible for re-weaving independent segments back into the original narrative network. Structured segment data is generated based on multiple text segments, the semantic tags and content summaries of each text segment, and the segment association pointers. Structured segment data refers to a comprehensive information representation entity with a standard logical architecture and context-aware capabilities, formed by deeply integrating the original physical text with its associated multidimensional business metadata, main idea summaries, and logical paths.

[0095] By performing refined semantic annotation and thematic extraction on each text segment, and reconstructing the logical connections between segments using segment association pointers, it is ensured that the segments can still carry and restore the narrative context and business orientation of the original document after physical separation. This encapsulation mechanism based on structured mapping effectively solves the problem of information silos, greatly improves the coherence and accuracy of the segmentation results in logical expression, and thus significantly improves the overall accuracy of text segmentation.

[0096] In this embodiment of the application, the step of inputting standardized text into the first large language model and outputting a set of semantic units includes: inputting standardized text into the first large language model and outputting a context-aware representation vector; guiding the first large language model to detect the semantic boundaries of the standardized text based on the context-aware representation vector through prompting engineering, and outputting a set of semantic units.

[0097] In this embodiment, the context-aware representation vector refers to the numerical representation entity generated after high-dimensional mapping of standardized text using a computing architecture with deep feature extraction capabilities. It not only carries the basic lexical meaning of the text but also deeply integrates the semantic relevance, logical dependence, and syntactic features of the character sequence in a specific context through an attention allocation mechanism, achieving a precise digital translation of the text's connotation. The first language model is guided by cue engineering to probe the semantic boundaries of standardized text based on the context-aware representation vector, outputting a set of semantic units. Cue engineering refers to the precise control and guidance of the reasoning behavior of the generative artificial intelligence architecture through the design, optimization, and combination of specific instruction templates, task constraints, or guiding examples, enabling it to perform complex logical recognition tasks according to predetermined business rules. The semantic boundary refers to the semantic watershed within a document where there is a natural switch, transition, or stage ending in the narrative theme, business logic, or intention expression; it represents the smallest dividing point in the text that cannot be further divided in the logical dimension.

[0098] By leveraging the largest language model to extract context-aware representation vectors with a global perspective, and supplementing this with cue engineering to target and stimulate the model's logical reasoning capabilities, fine-grained detection of standardized text semantic boundaries is achieved. This boundary recognition method, based on deep semantic connotations rather than physical length, effectively captures hidden logical turning points in documents, ensuring that the generated semantic unit sets are logically highly autonomous and have clear boundaries. This provides precise logical anchors for subsequent high-quality segmentation, significantly improving the overall accuracy of text segmentation.

[0099] Through the aforementioned technical solution, by multi-dimensionally perceiving standardized text, layout features, and the proportion of structured elements, the first major language model is used to deeply probe semantic boundaries and construct a global semantic relationship graph. This transforms traditional mechanical character stream segmentation into precise recognition based on logically autonomous units, thereby avoiding logical fragmentation caused by ignoring semantic coherence at the source. Furthermore, a dual verification is achieved through semantic coherence evaluation by the second major language model and dynamically generated segmentation constraint sets. This allows the segmentation action to adaptively adjust according to business tasks, document features, and logical tightness, ensuring that candidate segmentation positions are always anchored at the intersection of semantics and physical structure. Based on this, structured segmented data carrying semantic tags, content summaries, and association pointers is generated, and an internal coherence scoring mechanism based on the first major language model is introduced for closed-loop final review. This ensures that the final optimal segmented data, while complying with physical scale requirements, possesses extremely high logical self-consistency and contextual fidelity, thus significantly improving the accuracy of text segmentation.

[0100] Figure 2A schematic diagram illustrating the structure of a computer device according to an embodiment of this application is provided. Figure 2 As shown in the illustration, this application provides a computer device that may include: Memory 210 is configured to store instructions; and Processor 220 is configured to retrieve instructions from memory 210 and to implement the methods described above when executing instructions.

[0101] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the above-described method.

[0102] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0107] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0108] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0109] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0110] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A document adaptive fragmentation method, characterized in that, The method includes: Obtain standardized text, its corresponding set of typesetting features, and the proportion of structured data; The standardized text is input into the first large language model, and a set of semantic units is output. The semantic unit set is input into the first large language model, and a semantic relationship graph is output. The first set of proposed cutting points is determined based on the set of semantic units and the set of typesetting features; Based on the layout feature set, the proportion of structured data, the first set of proposed cutting points, the preset first scoring threshold, the semantic unit set, the fragmentation constraint set, and the second major language model, the standardized text is segmented to obtain multiple text fragments; Structured fragment data is generated based on the first large language model, the semantic relationship graph, and the multiple text fragments; The structured fragment data is used to generate an internal coherence score for the structured fragment data using the first large language model. If the internal coherence score is greater than a preset second score threshold, the structured fragment data is selected as the optimal fragment data.

2. The method according to claim 1, characterized in that, The steps for generating the fragmentation constraint set include: Obtain the downstream business task type; The corresponding business base coefficient, task complexity coefficient, and task length coefficient are determined based on the downstream business task type and the preset mapping table; the preset mapping table stores the mapping relationship between the business task type and the business base coefficient, task complexity coefficient, and task length coefficient. The total number of semantic units in the set of semantic units is taken as the total number of semantic units; The minimum number of semantic units is determined based on the total number of semantic units, the business fundamental coefficient, and the task complexity coefficient; The maximum character length is determined based on the proportion of structured data, the task length coefficient, and the preset basic character length. The fragmentation constraint set is generated based on the minimum number of semantic units, the maximum character length, and the semantic unit combination constraints.

3. The method according to claim 2, characterized in that, The step of segmenting the standardized text into multiple text segments based on the typesetting feature set, the proportion of structured data, the first set of proposed cutting points, the preset first scoring threshold, the semantic unit set, the segmentation constraint set, and the second major language model includes: Based on the layout feature set, the proportion of structured data, the standardized text, the semantic unit set, the first set of proposed cutting points, and the second large language model, the comprehensive semantic coherence score of each first proposed cutting point in the first set of proposed cutting points in the standardized text is determined. The second set of proposed cutting points is determined based on the comprehensive semantic coherence score of each first proposed cutting point in the standardized text and the preset first scoring threshold. Take each of the second proposed cutting points in the second set of proposed cutting points as the current cutting point, and repeat the following steps until all the second proposed cutting points in the second set of proposed cutting points have been traversed: The standardized text is segmented based on the current segmentation point and the previous segmentation point to obtain the candidate text interval between the two as a preliminary text segmentation. Determine the subset of semantic units corresponding to the preparatory text segment in the set of semantic units; If the semantic unit subset satisfies the segmentation constraint set, the current cutting point is taken as the valid cutting point; wherein, if the semantic unit subset satisfies the segmentation constraint set for the first time, the starting point of the standardized text is determined as the previous cutting point; Update the current cutting point to the previous cutting point, and update the next second proposed cutting point to the new current cutting point; The standardized text is segmented based on the effective cutting points and the starting points to obtain multiple text fragments.

4. The method according to claim 3, characterized in that, The comprehensive score for the semantic coherence of each first proposed cutting point in the standardized text, based on the layout feature set, the proportion of structured data, the standardized text, the semantic unit set, the first proposed cutting point set, and the second large language model, includes: The semantic coherence score is determined according to the following formula: in, A comprehensive score is given for the semantic coherence. The first text segment in the standardized text that ends at the first proposed cutting point; The second text segment in the standardized text, starting from the first proposed cutting point; The semantic connectivity score between the first text segment and the second text segment output by the second language model is given. The first type label is the semantic unit within the first semantic unit subset corresponding to the first intended cutting point in the semantic unit set of the first text fragment; The second type label is the semantic unit within the second semantic unit subset corresponding to the first proposed cutting point in the semantic unit set of the second text fragment; The business weight corresponding to the combined type label generated based on the first type label and the second type label in the preset weight allocation table; The fusion weight coefficient is obtained based on the proportion of the structured data. The typesetting feature matching score is obtained based on the position of the first proposed cutting point in the standardized text and the typesetting feature set.

5. The method according to claim 4, characterized in that, The semantic unit combination constraint satisfies the following formula: in, Constraints for the semantic unit combination; For the first semantic unit in the subset One semantic unit; For the first The type label of each semantic unit corresponds to the business weight in the preset weight allocation table; The preset weight threshold; The total number of semantic units in the semantic unit subset.

6. The method according to any one of claims 2 to 5, characterized in that, The process of determining the minimum number of semantic units based on the total number of semantic units, the business fundamental coefficient, and the task complexity coefficient includes: The minimum number of semantic units is determined according to the following formula: in, The minimum number of semantic units; The basic coefficient for the aforementioned business; The total number of semantic units; is the task complexity coefficient.

7. The method according to any one of claims 2 to 5, characterized in that, The step of determining the maximum character length based on the structured data ratio, the task length coefficient, and the preset basic character length includes: The maximum character length is determined according to the following formula: in, The maximum character length; The preset base character length; The proportion of the structured data; This is the task length coefficient.

8. The method according to any one of claims 1 to 5, characterized in that, The step of generating structured fragmented data based on the first large language model, the semantic relation graph, and the multiple text fragments includes: The first large language model is used to generate semantic tags and content summaries for each text segment; Based on the semantic relationship graph, a mapping relationship is established between each text segment to obtain the segment association pointer between each text segment; The structured fragment data is generated based on the multiple text fragments, the semantic tags and content summaries of each text fragment, and the fragment association pointers.

9. The method according to any one of claims 1 to 5, characterized in that, The step of inputting the standardized text into the first large language model and outputting a set of semantic units includes: The standardized text is input into the first language model, and the context-aware representation vector is output. The prompting engineering guides the first large language model to detect the semantic boundaries of the standardized text based on the context-aware representation vector, and outputs a set of semantic units.

10. A computer device, characterized in that, include: The memory is configured to store instructions; as well as A processor configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method according to any one of claims 1 to 9.

Images