Task branching control method and system driven by pre-analysis and pre-processing of large language model

Abstract

The invention relates to the technical field of electric digital data processing, and discloses a task branching control method and system driven by pre-analysis and pre-processing of a large language model, and the method comprises the steps: analyzing a to-be-processed document set, generating semantic data blocks, and distributing an index identifier with a hierarchical relationship for each semantic data block; performing feature extraction on the semantic data blocks by adopting a pre-constructed feature extraction model to obtain a multi-dimensional feature set so as to construct a structured index table, and dynamically retrieving a candidate data block list from the structured index table according to a received user query task; generating a data block loading strategy according to the multi-dimensional feature set associated with each candidate data block in the candidate data block list; and according to a data block loading strategy, performing branching control on processing of the selected quantized large language model on the candidate data blocks to obtain a task processing result. According to the method provided by the invention, the processing capability of the quantized large language model on complex and long document tasks is improved, and the computing power consumption is reduced.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a task-based control method and system driven by large language model pre-analysis and pre-processing. Background Technology

[0002] With the breakthrough development of large language model technology, it has been widely used in document understanding, information retrieval, and intelligent question answering. However, the operation of large language models relies on the support of high-performance computing hardware, and enterprise computing cards cannot meet its requirements. Existing technologies reduce the storage space occupation and computing resource consumption of the model by performing operations such as weight quantization, pruning, and knowledge distillation on the original large language model, while ensuring that the core capabilities of the model are basically not damaged, so that it can be deployed with the support of a small number of computing cards.

[0003] However, when applying the above-mentioned quantized large language model to the processing of massive and complex unstructured documents in enterprises, a series of severe technical bottlenecks are encountered: model performance loss leads to insufficient ability to process complex tasks; the context window of the quantized large language model is limited and cannot be adapted to long text office scenarios; insufficient computing power leads to poor multi-task concurrent processing capability.

[0004] Therefore, improving the ability of large quantized language models to handle complex and long document tasks, and reducing computational consumption have become urgent technical problems for those skilled in the art. Summary of the Invention

[0005] This invention provides a task-based control method and system driven by large language model pre-analysis and preprocessing to address the technical problems of how to improve the processing capability of large quantized language models for complex and long document tasks, and how to reduce computing power consumption.

[0006] In a first aspect, the present invention provides a task-based control method driven by large language model pre-analysis and preprocessing, the method comprising: A multi-granularity adaptive semantic segmentation algorithm is used to parse the document set to be processed, generate semantic data blocks, and assign an index identifier with hierarchical relationship to each semantic data block; A pre-built feature extraction model is used to extract features from each of the semantic data blocks to obtain a multi-dimensional feature set. Based on the semantic data block, the index identifier, and the multidimensional feature set, a structured index table is constructed, and a candidate data block list is obtained by dynamically retrieving from the structured index table according to the received user query task. A data block loading strategy is generated based on the multidimensional feature set associated with each candidate data block in the candidate data block list. The data block loading strategy is set to be determined based on the task adaptability and topic relevance between the multidimensional feature set and the user query task, as well as the hierarchical relationship between the index identifiers. According to the data block loading strategy, the selected quantized large language model is controlled to process the candidate data blocks, generate intermediate processing results corresponding to each candidate data block, and integrate the intermediate processing results according to the corresponding index identifier to obtain the task processing result.

[0007] Preferably, the step of using a multi-granularity adaptive semantic segmentation algorithm to parse the document set to be processed and generate semantic data blocks includes: Multi-granularity scanning is performed on the documents to be processed in the document set to identify candidate segmentation points at different granularity levels and obtain a sequence of candidate segmentation points. Calculate the topic consistency score of the text units on both sides of each candidate segmentation point, and select the optimal segmentation point sequence from the candidate segmentation point sequence based on the topic consistency score and a preset adaptive threshold. The adaptive threshold is dynamically calculated based on at least one of document type, text length and domain keyword density. Based on the optimal segmentation point sequence, semantic segmentation is performed on the document to be processed, and a hierarchical relationship between the segmentation blocks is constructed according to the granular nesting relationship in the semantic segmentation process, to obtain semantic data blocks with hierarchical relationships.

[0008] Preferably, the dynamic calculation method includes: An adaptive threshold calculation formula is obtained by fitting the document type, text length, domain keyword density, and topic consistency score of manually annotated segmented documents with different granularity segmentation points. Based on the document type, paragraph length, and domain keyword density corresponding to the document to be processed, the adaptive threshold corresponding to the document to be processed is obtained by using the adaptive threshold calculation formula.

[0009] Preferably, the pre-built feature extraction model is used to extract features from each semantic data block to obtain a multi-dimensional feature set, including: A pre-built feature extraction model is used to process each semantic data block to obtain the topic vector, task adaptation type label, and cross-block correlation matrix of the semantic data block; the feature extraction model includes a semantic encoder and a multi-task output head; A pre-built risk assessment engine is used to perform data conflict detection on the topic vector to obtain the data conflict risk level of the semantic data block; Based on the topic vector, the task adaptation type label, the cross-block correlation matrix, and the data conflict risk level, a multi-dimensional feature set of the semantic data block is constructed.

[0010] Preferably, the structured index table has a hierarchical cascaded architecture and includes at least: The task adaptation sub-index layer is constructed based on the index identifier and the task adaptation type label corresponding to each semantic data block, and is used to realize fast filtering based on task type. The semantic adaptation sub-index layer is constructed based on the index identifier and topic vector corresponding to each semantic data block, and is used to realize topic similarity-based retrieval. The context-related sub-index layer is constructed based on the index identifiers corresponding to each semantic data block and the cross-block correlation matrix, and is used to support the retrieval of related contexts.

[0011] Preferably, the step of dynamically retrieving a candidate data block list from the structured index table based on the received user query task includes: The received user query task is parsed to obtain the query task type, semantic query vector, and context association requirements; Based on the query task type, a search is performed in the task adaptation sub-index layer to obtain the first candidate set; Based on the semantic query vector, a similarity search is performed on the first candidate set in the semantic adaptation sub-index layer to obtain the second candidate set; Based on the aforementioned context association requirements, the elements in the second candidate set are associated and expanded in the context association sub-index layer to obtain a third candidate set. A list of candidate data blocks is obtained based on the semantic data blocks associated with the second candidate set and the semantic data blocks associated with the third candidate set.

[0012] Preferably, generating a data block loading strategy based on the multidimensional feature set associated with each candidate data block in the candidate data block list includes: Based on the multidimensional feature set associated with each candidate data block in the candidate data block list, the task fit and topic relevance between each multidimensional feature set and the user query task are obtained; based on the data conflict risk level associated with the candidate data block, the risk penalty coefficient is obtained. A comprehensive priority score is obtained based on the task suitability, the topic relevance, and the risk penalty coefficient. Based on the hierarchical relationship between each index identifier, an initial loading queue is obtained, and based on the comprehensive priority score, the candidate data blocks at the same level of the initial loading queue are adjusted to obtain a data block loading strategy.

[0013] Preferably, the step of controlling the processing of the candidate data blocks by the selected quantized large language model according to the data block loading strategy, and generating intermediate processing results corresponding to each candidate data block, includes: Based on the context window length limit of the selected quantized large language model, set a fixed capacity for the sliding window controller; According to the data block loading strategy, the contents of the corresponding candidate data blocks are filled into the current sliding window of the sliding window controller, and a data block length check is performed during the content filling process to ensure that the total length of the current sliding window is less than or equal to the fixed capacity. Perform a secondary semantic split on a single candidate data block whose length exceeds the fixed capacity, and assign an extended index identifier to the split sub-semantic data blocks; The content of the current sliding window is input into the quantized large language model to generate an intermediate processing result associated with the index identifier or the extended index identifier.

[0014] Preferably, the step of integrating the intermediate processing results according to the corresponding index identifier to obtain the task processing result includes: Based on the hierarchical relationship of the index identifier or the extended index identifier associated with the intermediate processing results, the intermediate processing results are integrated to obtain the initial task processing results; Based on the data conflict risk level of the semantic data block associated with the intermediate processing result, the initial task processing result is semantically conflict corrected to obtain the task processing result.

[0015] Secondly, the present invention also provides a task splitting control system driven by large language model pre-analysis and preprocessing, for implementing the task splitting control method driven by large language model pre-analysis and preprocessing described above. The system includes: a document parsing module, a feature extraction module, a data block retrieval module, a loading strategy generation module, and a task processing module. The document parsing module is used to parse the document set to be processed using a multi-granularity adaptive semantic segmentation algorithm, generate semantic data blocks, and assign an index identifier with a hierarchical relationship to each semantic data block. The feature extraction module is used to extract features from each of the semantic data blocks using a pre-built feature extraction model to obtain a multi-dimensional feature set. The data block retrieval module is used to construct a structured index table based on the semantic data block, the index identifier, and the multidimensional feature set, and dynamically retrieve a candidate data block list from the structured index table according to the received user query task. The loading strategy generation module is used to generate a data block loading strategy based on the multidimensional feature set associated with each candidate data block in the candidate data block list. The data block loading strategy is set to be determined based on the task adaptability and topic relevance between the multidimensional feature set and the user query task, as well as the hierarchical relationship between the index identifiers. The task processing module is used to control the selected quantized large language model to process the candidate data blocks according to the data block loading strategy, generate intermediate processing results corresponding to each candidate data block, and integrate the intermediate processing results according to the corresponding index identifier to obtain the task processing result.

[0016] This invention provides a task-based control method and system driven by large language model pre-analysis and preprocessing. Compared with existing technologies, the embodiments of this invention have the following advantages: The task-oriented control method driven by pre-analysis and preprocessing of large language models provided in this application breaks through the context and computing power limitations of quantitative large language models, realizing the full-process analysis of long documents. Through a combination of semantic segmentation and dynamic loading strategies, it solves the problem that existing quantitative models cannot handle long documents due to limited context windows. At the same time, through precise screening and loading of semantic data blocks, it reduces the computational consumption of invalid data and lowers the dependence on computing power, enabling quantitative large language models with a small number of computing cards to efficiently complete the entire analysis task of long documents, improving the adaptability of quantitative large language models in complex office scenarios. The structured processing and index binding of the entire process greatly improve the efficiency of semantic data block retrieval and semantic data block loading. Combined with key feature extraction and risk assessment, quantitative large language models can more accurately grasp the core requirements of the task, reduce invalid processing, and improve the processing efficiency of intelligent office tasks. Semantic conflict correction in the result integration stage further ensures the integrity and accuracy of the output results, effectively solving the problems of insufficient complex task processing capabilities and chaotic output logic in existing technologies. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the steps of a task-based control method driven by large language model pre-analysis and preprocessing, provided in a preferred embodiment of the present invention. Figure 2 This is a schematic diagram of a task-based control system driven by large language model pre-analysis and preprocessing, provided in a preferred embodiment of the present invention. Figure Labels 1-Document parsing module, 2-Feature extraction module, 3-Data block retrieval module, 4-Loading strategy generation module, 5-Task processing module. Detailed Implementation

[0018] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are provided for illustrative purposes only and should not be construed as limiting the scope of the invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of protection of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of this invention.

[0019] In the description of this invention, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0020] Please see Figure 1 In an embodiment of the present invention, a task-based control method driven by large language model pre-analysis and preprocessing is provided, the method comprising: S1. A multi-granularity adaptive semantic segmentation algorithm is used to parse the document set to be processed, generating semantic data blocks, and assigning an index identifier with hierarchical relationship to each semantic data block. In a preferred embodiment of this application, the document set to be processed is obtained, including documents of various formats. The documents to be processed are subjected to format standardization processing, such as unifying the encoding format, removing invalid format tags, and repairing damaged document structures, to ensure the stability of subsequent document parsing. Further, the documents to be processed are scanned at multiple granularities to initially find possible segmentation boundaries in the documents at different semantic scales, obtaining a sequence of candidate segmentation points. Specifically, a set of granularity levels is predefined, such as chapter level, paragraph level, sentence level, clause level, etc., with each level corresponding to a different scanning window and segmentation unit. At the chapter level, explicit markers such as "Chapter X" and "## Title" are scanned; at the paragraph level, line breaks or indentation are identified; at the finer-grained sentence and clause levels, pre-trained sequence labeling models are used to identify potential segmentation locations. These sequence labeling models are based on deep learning neural network models and can identify potential weak boundaries such as commas, semicolons, and periods. For example, the BiLSTM-CRF model (Bidirectional Long Short-Term Memory Network-Conditional Random Field model) can perform named entity recognition, part-of-speech tagging, and word segmentation.

[0021] Furthermore, the topic consistency score of the text units on both sides of each candidate segmentation point is calculated to quantify the semantic coherence of the text units on both sides of the candidate segmentation point. The lower the topic consistency score, the greater the topic difference between the text units on both sides of the candidate segmentation point, and the more suitable it is to segment at this point. Specifically, a certain length of text is truncated to the left and right of the candidate segmentation point, respectively, as the left and right text units, with the truncated length varying with the granularity level. The left and right text units are encoded using a pre-trained language model, resulting in fixed-dimensional left and right semantic vectors. The cosine similarity between the left and right semantic vectors is calculated to obtain the topic consistency score between the left and right semantic vectors. The higher the topic consistency score, the closer the text units on both sides of the candidate segmentation point are in the semantic space, the more consistent their topics, and the less suitable it is to segment at this point; the lower the topic consistency score, the farther apart the text units on both sides of the candidate segmentation point are in the semantic space, the more inconsistent their topics, and the more suitable it is to segment at this point.

[0022] Furthermore, based on the topic consistency score and a preset adaptive threshold, the optimal segmentation point sequence is selected from the candidate segmentation points. Specifically, for all candidate segmentation points at a certain granularity level, their topic consistency scores are compared with the preset adaptive threshold, and candidate segmentation points with topic consistency scores lower than the adaptive threshold are selected as the optimal segmentation point sequence.

[0023] In a preferred embodiment of this application, the adaptive threshold is dynamically calculated based on at least one of document type, text length, and domain keyword density. Specifically, the document type, text length, domain keyword density, and topic consistency scores of different granularity segmentation points of the manually annotated segmented documents are fitted to obtain the adaptive threshold calculation formula. The adaptive threshold calculation formula obtained in this application is expressed as follows: in, Indicates particle size Adaptive threshold, Indicates particle size The basic threshold, Indicates particle size Adjust the weight of document types. Indicates particle size The document type adjustment factor is preset according to the document type. Indicates particle size Adjusting the weight of text length Indicates particle size The text length adjustment factor is preset based on the text length. Indicates particle size Adjust the weight of keyword density in the field. Indicates particle size The domain keyword density adjustment factor is preset based on the domain keyword density. The document type adjustment weight, text length adjustment weight, and domain keyword density adjustment weight are obtained through fitting.

[0024] Based on the document type, text length, and domain keyword density of the document to be processed, determine the document type adjustment factor, text length adjustment factor, and domain keyword density adjustment factor of the document to be processed. Substitute these factors into the adaptive threshold calculation formula to obtain the adaptive threshold for the document to be processed.

[0025] Furthermore, the actual semantic segmentation operation is performed based on the optimal segmentation point. Following a coarse-to-fine granularity order, the document to be processed is divided into independent segmentation blocks at the optimal segmentation point. For example, chapters are determined first, and then paragraphs are determined within each chapter. During semantic segmentation, the segmentation blocks generated by the coarse-grained segmentation automatically become the parent blocks of the segmentation blocks generated by the fine-grained segmentation within them. For example, a "chapter block" is the parent block of all its "paragraph blocks". Under the same parent block, child blocks arranged in document order are sibling blocks. The semantic association between different segmentation blocks is calculated as the weight between different segmentation blocks, i.e., the association relationship between different segmentation blocks. Finally, a semantic data block tree is generated, with each node representing a semantic data block.

[0026] Furthermore, a unique index identifier is assigned to each semantic data block. Specifically, a root identifier is assigned to each document. Starting from the root identifier, a depth-first traversal is used. Whenever a child block is created, a sequence number is appended to its parent block identifier. For example, if the root identifier of the document is D1, the first-level blocks are D1.1 and D1.2, the second-level blocks are D1.1.1, D1.1.2, and D1.2.1, and the leaf block is D1.1.1.1.

[0027] In the preferred embodiment of this application, by calculating topic consistency scores and applying adaptive thresholds, it is ensured that each segmentation point lies on the semantic boundary, making each generated semantic data block as semantically complete and independent as possible. This provides high-quality input units for the subsequent accurate understanding of large language models. The generated semantic data blocks are no longer plain text fragments, but standardized objects with unique identifiers and hierarchical relationships, making them independently addressable and associative knowledge atoms, providing core support for subsequent accurate retrieval, ordered loading, and result integration.

[0028] S2. Using a pre-built feature extraction model, features are extracted from each semantic data block to obtain a multi-dimensional feature set. In a preferred embodiment of this application, after obtaining semantic data blocks with hierarchical relationships, a pre-built feature extraction model is used to deeply understand the semantic connotation and functional attributes of each semantic data block and transform them into structured digital features that can be directly processed, analyzed, and calculated by the machine. Specifically, the feature extraction model includes a semantic encoder and a multi-task output head. The semantic encoder is built based on a pre-trained language model and is responsible for extracting rich contextual semantic representations from the semantic data blocks to obtain feature vectors. Above the semantic encoder, multiple specific task output layers are connected. In this application, the task output layers include a topic vector output layer, a task adaptation type output layer, and a cross-block correlation output layer. The topic vector output layer is a fully connected projection layer used to map feature vectors to topic vectors; the task adaptation type output layer is a fully connected layer and a classifier using a sigmoid function (S-shaped function) used to predict task adaptation type labels; the cross-block correlation output layer is a self-attention-based interactive encoding layer used to output the cross-block correlation matrix.

[0029] Furthermore, following the feature extraction model, a pre-built risk assessment engine is integrated. This engine is configured to simultaneously evaluate the consistency between the topic vectors of semantic data blocks and their logically related parts within the document, thereby identifying potential data conflict risks. Specifically, for the semantic data block to be evaluated, related semantic data blocks that should maintain a high degree of logical consistency are determined based on their hierarchical relationship. As a preferred embodiment, the related semantic data blocks include at least: the block's direct parent block and its direct predecessor and / or successor sibling blocks. The risk assessment engine receives the topic vectors output from the feature extraction model and calculates the cosine similarity between the topic vectors of the current semantic data block to be evaluated and the topic vectors of the related semantic data blocks to obtain a vector similarity value. Within the same document, the topic vectors of the current semantic data block and the related semantic data blocks should ideally maintain a high degree of consistency and have a high vector similarity value. If the vector similarity value is lower than a pre-set similarity threshold, it indicates that the corresponding semantic data blocks are severely deviating in semantic space. Therefore, in this application, the data conflict risk level of the semantic data block is determined based on the degree to which the vector similarity value is lower than the similarity threshold. Specifically, using the gradient mapping method, a set of difference ranges between vector similarity values ​​and similarity thresholds are predefined, along with the data conflict risk levels corresponding to these difference ranges, forming a risk level mapping table. An example of the risk level mapping table is shown below: The range of differences between the similarity threshold and the vector similarity value is determined, and the corresponding data conflict risk level is determined according to the data conflict risk level mapping table. If the semantic data block to be evaluated deviates from multiple related blocks, the highest data conflict risk level is taken as the final data conflict risk level of the semantic data block to be evaluated.

[0030] Finally, a multidimensional feature set of semantic data blocks is constructed based on topic vectors, task adaptation type labels, cross-block correlation matrix, and data conflict risk level.

[0031] In a preferred embodiment of this application, by integrating deep semantic parsing with synchronous risk perception, the generated multidimensional feature set not only describes the semantics, functions, and relationships of the data block, but also incorporates quality diagnostic information on its own consistency. This achieves a qualitative change from feature description to feature diagnosis, providing data support for subsequent semantic conflict modification of the intermediate processing results output by the quantized large language model, and significantly improving the accuracy and reliability of the final task processing results.

[0032] S3. Based on the semantic data block, the index identifier, and the multidimensional feature set, a structured index table is constructed, and a candidate data block list is dynamically retrieved from the structured index table according to the received user query task. In a preferred embodiment of this application, the structured index table is not a single data table, but a composite index system with a hierarchical and cascaded architecture. It consists of multiple complementary and interconnected sub-index layers, which together efficiently organize the different dimensional features of the semantic data block to support subsequent complex multimodal retrieval. The structured index table includes at least: a task adaptation sub-index layer, a semantic adaptation sub-index layer, and a context association sub-index layer. The task adaptation sub-index layer is constructed based on the index identifier and task adaptation type label corresponding to each semantic data block. Its construction process includes: building an inverted index list for each occurrence of a task type label; when a semantic data block is assigned one or more task type labels, its index identifier is inserted into the list of the corresponding task type labels; the task adaptation sub-index layer is used to implement Boolean filtering based on task type and can return a set of index identifiers for all semantic data blocks belonging to a specific task type. The semantic adaptation sub-index layer is constructed based on the index identifiers and topic vectors corresponding to each semantic data block. Its construction process includes importing the index identifiers and topic vectors corresponding to each semantic data block into a dedicated vector database. In a preferred embodiment of this application, the HNSW (Hierarchical Navigable Small World) algorithm is used to construct the index. The semantic adaptation sub-index layer can return the index identifiers of the top-K semantically similar semantic data blocks based on a given query vector. The context association sub-index layer is constructed based on the index identifiers and cross-block association degree matrix corresponding to each semantic data block. Its construction process includes treating each semantic data block as a graph node, using association relationships in the cross-block association degree matrix greater than a preset association threshold as undirected edges connecting two graph nodes, with the edge weight representing the association degree. The final graph structure is stored in a graph database. The context association sub-index layer can quickly find semantically closely related neighboring semantic data blocks based on a given index identifier.

[0033] In a preferred embodiment of this application, all sub-index layers are associated and traced back through the index identifier of semantic data blocks. At the same time, a central metadata registry is maintained. This central metadata registry uses the index identifier as the primary key and stores complete information such as the original content, hierarchical relationship, topic vector, task adaptation type label, and data conflict risk level of each semantic data block, ensuring that the index identifier retrieved from any sub-index layer can quickly locate the complete semantic data block information.

[0034] Furthermore, the system receives query tasks input by users in natural language, keywords, or a combination of these formats. A pre-built query parsing model parses the user's query task to generate structured query conditions. The parsing process includes: identifying the user's intent, determining whether the user needs factual answers, concept explanations, or content summaries, to generate the query task type; performing core semantic recognition on the user's query task, converting the obtained core semantics into a query vector using the same semantic encoder as the feature extraction, resulting in the semantic query vector of the user's query task; and performing contextual intent recognition on the user's query task to obtain the contextual requirements of the user's query task. In a preferred embodiment of this application, contextual intent recognition is implemented based on a pre-built contextual classifier. The contextual classifier uses a lightweight text classification model as its infrastructure and is trained using a training dataset labeled according to pre-built association rules. In practical applications, the user's query task is input into the trained contextual classifier, which outputs the corresponding contextual requirements. An example of an association rule in this preferred embodiment is as follows: Furthermore, based on the query task type, semantic query vector, and contextual requirements, structured query conditions are formed. The retrieval engine then performs a dynamic, multi-level retrieval process on the structured index table based on these structured query conditions. This process typically follows a cascading logic of "first filtering, then detailed searching, and then expansion". Specifically, the process first accesses the task adaptation sub-index layer. Based on the query task type in the structured query conditions, it retrieves all matching index identifier sets as the first candidate set, quickly eliminating a large amount of irrelevant content. Next, it accesses the semantic adaptation sub-index layer. Based on the semantic query vector in the structured query conditions, it performs a similarity search on the first candidate set to obtain the second candidate set. This limits the search scope to the vector subspace corresponding to the first candidate set obtained in the previous step, significantly reducing the computational scope and improving retrieval speed and accuracy. Further, if the structured query conditions specify the need for context association, it accesses the context association sub-index layer. For each index identifier in the second candidate set, it searches for its strongly associated neighbor nodes in the graph structure, determining the index identifiers of these neighbor nodes as the third candidate set. Finally, it uses the semantic data blocks associated with the second candidate set as the core candidate data block set and the semantic data blocks associated with the third candidate set as the supplementary candidate data block set. Based on the core and supplementary candidate data block sets, a candidate data block list is constructed.

[0035] In the preferred embodiment of this application, a cascading retrieval strategy of "first screening, then detailed searching, and then expansion" is adopted, which not only ensures extremely high precision but also achieves an order-of-magnitude improvement in retrieval efficiency, thus solving the speed bottleneck problem of semantic retrieval under massive vector data.

[0036] S4. Based on the multidimensional feature set associated with each candidate data block in the candidate data block list, a data block loading strategy is generated. The data block loading strategy is set to be determined based on the task adaptability and topic relevance between the multidimensional feature set and the user query task, as well as the hierarchical relationship between the index identifiers. In a preferred embodiment of this application, based on the multidimensional feature set associated with each candidate data block in the candidate data block list, the task adaptability and topic relevance between each multidimensional feature set and the user query task are obtained. The task adaptability is an indicator used to quantify the degree of matching between the task adaptability type label of the semantic data block and the query task type of the user query task. It is calculated based on a first matching degree mapping function, which is expressed as: in, Represents candidate data blocks User query task Task compatibility between them Represents a candidate data block. This indicates that the user is querying a task. Represents candidate data blocks The set of associated task adaptation type tags, Indicates the first Task adaptation type tags Indicates user query task The collection of query task types parsed from the data. This indicates an indicator function that appears when the task matches the type label. A collection of query task types When the time condition is met, its value is 1; otherwise, it is 0. Indicates task adaptation type tag The preset weight is determined based on the importance of the task adaptation type tag in the business scenario.

[0037] Topic relevance is a metric used to quantify the degree of matching between the topic vector of a semantic data block and the semantic query vector of a user query task. It is calculated based on a second matching degree mapping function, which is expressed as: in, Represents candidate data blocks User query task The degree of thematic relevance between them Represents candidate data blocks Theme vectors, User query task The semantic query vector parsed from the data.

[0038] Furthermore, based on the data conflict risk level associated with the candidate data block, a risk penalty coefficient is obtained. This risk penalty coefficient is a factor used to apply negative adjustment in the loading decision. The higher the data conflict risk level, the larger the calculated risk penalty coefficient, thereby reducing the ranking position of the candidate data block in the subsequent comprehensive priority calculation. In a preferred embodiment of this application, an exponential penalty formula is used to map the discrete data conflict risk level to the risk penalty coefficient, which is then expressed as: in, Indicates the level of data conflict risk. The corresponding risk penalty coefficient, Represents candidate data blocks Data conflict risk level Indicates the base penalty coefficient. , Indicates growth factor, Control the severity of penalties as the risk level of data conflict increases. Represents the natural constant. This represents a function that quantifies the level of data conflict risk into a numerical value. For example, if the level of data conflict risk is {low, medium, high}, it can be mapped to {0, 1, 2} or {0, 0.5, 1.0}.

[0039] Furthermore, a weighted sum is calculated based on task suitability, topic relevance, and risk penalty coefficient to obtain a comprehensive priority score, which is expressed as follows: in, Represents candidate data blocks The overall priority score, The weighting coefficients represent the task suitability. Weighting coefficients representing topic relevance. The weighting coefficient representing the risk penalty coefficient is pre-set based on task suitability, topic relevance, and the importance of the risk penalty coefficient.

[0040] After calculating the overall priority score of each candidate data block, an initial loading queue is obtained based on the hierarchical relationship between the index identifiers corresponding to each candidate data block. This initial loading queue satisfies that at any given time, a parent block must have already finished loading or be loading before all its child blocks begin loading. However, the order of sibling blocks under the same parent block, i.e., semantic data blocks at the same level, in the queue is determined solely by their order of appearance in the original document, without considering their overall priority. After obtaining the initial loading queue that satisfies the hierarchical constraints, all consecutive semantic data block sequences with the same direct parent block are located and processed. These sequences are called "same-level block groups." For each "same-level block group," the candidate data blocks are reordered in descending order within the group based on their overall priority scores. This adjustment is strictly limited to within the "same-level block group," ensuring that the global hierarchical constraints are still fully satisfied after adjustment. The resulting optimal loading queue maintains the global hierarchical logic of parent-to-child and also achieves local optimization within each group of sibling blocks, loading from high to low overall priority. By encapsulating the optimal loading queue into a data block loading strategy, and prioritizing high-value, highly adaptable content while respecting cognitive logic, the output quality of quantization large language model tasks is significantly improved.

[0041] S5. According to the data block loading strategy, the selected quantized large language model is controlled to process the candidate data blocks, generating intermediate processing results corresponding to each candidate data block, and integrating the intermediate processing results according to the corresponding index identifier to obtain the task processing result. In this limited embodiment of the application, after obtaining the optimized data block loading strategy, the challenge is how to efficiently process candidate data block sequences that may far exceed the length of a single context window using the large language model. To this end, in a preferred embodiment of the application, semantic data blocks are loaded through a sliding window controller, the fixed capacity of which is determined according to the context window length limit of the selected quantized large language model. The sliding window controller reads the data block loading strategy and obtains a list of sequentially arranged data blocks to be processed. Following this list, the controller sequentially retrieves the next candidate data block from the queue and adds it to the current sliding window. During this process, a data block length check is performed. After adding the candidate data block, it is determined whether the total length of the current sliding window is less than or equal to the fixed capacity. If so, the candidate data block is added to the current sliding window; otherwise, the addition of data blocks to the current sliding window is stopped. On the other hand, it checks whether the length of a single candidate data block is greater than the fixed capacity. If so, a secondary semantic split is performed on the candidate data block, using a finer-grained semantic segmentation algorithm to split it into several sub-semantic data blocks. Each sub-semantic data block is assigned an extended index identifier. For example, if the original index identifier of the unprocessed semantic data block is D1.001.055, then the extended index identifiers of the sub-semantic data blocks are D1.001.055.1 and D1.001.055.2 to maintain traceability. Then, the first sub-semantic data block is added to the current sliding window. Once the current sliding window is full, the content of the current sliding window is submitted to the quantized large language model to generate intermediate processing results. These intermediate processing results are associated with the index identifier of the corresponding semantic data block or the extended index identifier of the sub-semantic data block.

[0042] After processing a sliding window, an overlapping sliding strategy is adopted to retain a certain length of content at the end of the previous sliding window as the starting context of the new sliding window. This ensures that the quantized large language model can remember key recent historical information when processing new content, thus guaranteeing the continuity of intermediate processing results.

[0043] Furthermore, based on the hierarchical relationship of the index identifiers or extended index identifiers associated with the intermediate processing results, the intermediate processing results are integrated to obtain the initial task processing results. For example, for extraction tasks, the lists are directly merged and deduplicated; for generation tasks, a text fusion strategy is adopted to ensure the logical coherence of the statements.

[0044] On the other hand, in the process of large language model processing, inconsistencies may exist between different semantic data blocks, causing some semantic conflicts in the initial task processing results. In the preferred embodiment of this application, data conflict risk is used as a key clue to automatically correct the initial task processing results, so as to proactively identify and resolve potential contradictions in the initial task processing results, and ensure that the final generated task processing results are self-consistent, reliable, and of high quality. Specifically, based on the data conflict risk level, intermediate processing results with high risk levels are identified and determined as high-risk intermediate processing results. It is then determined whether there is any content in the high-risk processing results that is inconsistent with other intermediate processing results. If so, the content of other intermediate processing results is used to replace the corresponding content in the high-risk processing results to improve the accuracy of the task processing results.

[0045] In the preferred embodiment of this application, the fundamental contradiction between the limited context window of a large quantized language model and the unlimited demand for long document processing is resolved through window scheduling and streaming processing, achieving the effect of reliably processing documents of arbitrary length using a fixed window model. By utilizing data conflict risk levels, the initial task processing results are automatically corrected, improving the accuracy of the final task processing results.

[0046] Accordingly, such as Figure 2 As shown, based on a task splitting control method driven by large language model pre-analysis and preprocessing, this application embodiment also provides a task splitting control system driven by large language model pre-analysis and preprocessing, used to implement the task splitting control method driven by large language model pre-analysis and preprocessing disclosed in this application embodiment. The system includes: a document parsing module 1, a feature extraction module 2, a data block retrieval module 3, a loading strategy generation module 4, and a task processing module 5. The document parsing module 1 is used to parse the document set to be processed using a multi-granularity adaptive semantic segmentation algorithm, generate semantic data blocks, and assign an index identifier with a hierarchical relationship to each semantic data block. The feature extraction module 2 is used to extract features from each semantic data block using a pre-built feature extraction model to obtain a multi-dimensional feature set. The data block retrieval module 3 is used to construct a structured index table based on the semantic data block, the index identifier, and the multidimensional feature set, and dynamically retrieve a candidate data block list from the structured index table according to the received user query task. The loading strategy generation module 4 is used to generate a data block loading strategy based on the multidimensional feature set associated with each candidate data block in the candidate data block list. The data block loading strategy is set to be determined based on the task adaptability and topic relevance between the multidimensional feature set and the user query task, as well as the hierarchical relationship between the index identifiers. The task processing module 5 is used to control the selected quantized large language model to process the candidate data blocks according to the data block loading strategy, generate intermediate processing results corresponding to each candidate data block, and integrate the intermediate processing results according to the corresponding index identifier to obtain the task processing result.

[0047] Specific limitations regarding the task-based control system driven by large language model pre-analysis and pre-processing can be found in the above-described limitations of the task-based control method driven by large language model pre-analysis and pre-processing, and will not be repeated here. Those skilled in the art will recognize that the various modules and steps described in conjunction with the embodiments disclosed in this invention can be implemented in hardware, software, or a combination of both. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0048] This embodiment provides a task-based control method and system driven by large language model pre-analysis and preprocessing, which is used to solve the technical problems of how to improve the processing capability of large quantized language models for complex tasks and long document tasks, and reduce computing power consumption. The method includes: parsing the document set to be processed using a multi-granularity adaptive semantic segmentation algorithm to generate semantic data blocks, and assigning an index identifier with a hierarchical relationship to each semantic data block; extracting features from the semantic data blocks using a pre-built feature extraction model to obtain a multi-dimensional feature set; constructing a structured index table based on the semantic data blocks, index identifiers, and multi-dimensional feature sets, and dynamically retrieving a list of candidate data blocks from the structured index table according to the received user query task; generating a data block loading strategy based on the multi-dimensional feature set associated with each candidate data block in the candidate data block list, wherein the data block loading strategy is set to be determined based on the task adaptability and topic relevance between the multi-dimensional feature set and the user query task, as well as the hierarchical relationship between the index identifiers; and, according to the data block loading strategy, controlling the selected quantized large language model to process the candidate data blocks, generating intermediate processing results corresponding to each candidate data block, and integrating the intermediate processing results according to the corresponding index identifiers to obtain the task processing result. The task-oriented control method driven by pre-analysis and preprocessing of large language models provided in this application breaks through the context and computing power limitations of quantitative large language models, realizing the full-process analysis of long documents. Through a combination of semantic segmentation and dynamic loading strategies, it solves the problem that existing quantitative models cannot handle long documents due to limited context windows. At the same time, through precise screening and loading of semantic data blocks, it reduces the computational consumption of invalid data and lowers the dependence on computing power, enabling quantitative large language models with a small number of computing cards to efficiently complete the entire analysis task of long documents, improving the adaptability of quantitative large language models in complex office scenarios. The structured processing and index binding of the entire process greatly improve the efficiency of semantic data block retrieval and semantic data block loading. Combined with key feature extraction and risk assessment, quantitative large language models can more accurately grasp the core requirements of the task, reduce invalid processing, and improve the processing efficiency of intelligent office tasks. Semantic conflict correction in the result integration stage further ensures the integrity and accuracy of the output results, effectively solving the problems of insufficient complex task processing capabilities and chaotic output logic in existing technologies.

[0049] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts of the embodiments, refer to each other. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0050] The above-described embodiments are merely preferred embodiments of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A task-based control method driven by large language model pre-analysis and preprocessing, characterized in that, The method includes: A multi-granularity adaptive semantic segmentation algorithm is used to parse the document set to be processed, generate semantic data blocks, and assign an index identifier with hierarchical relationship to each semantic data block; A pre-built feature extraction model is used to extract features from each of the semantic data blocks to obtain a multi-dimensional feature set. Based on the semantic data block, the index identifier, and the multidimensional feature set, a structured index table is constructed, and a candidate data block list is obtained by dynamically retrieving from the structured index table according to the received user query task. A data block loading strategy is generated based on the multidimensional feature set associated with each candidate data block in the candidate data block list. The data block loading strategy is set to be determined based on the task adaptability and topic relevance between the multidimensional feature set and the user query task, as well as the hierarchical relationship between the index identifiers. According to the data block loading strategy, the selected quantized large language model is controlled to process the candidate data blocks, generate intermediate processing results corresponding to each candidate data block, and integrate the intermediate processing results according to the corresponding index identifier to obtain the task processing result.

2. The task-based control method driven by large language model pre-analysis and preprocessing as described in claim 1, characterized in that, The process employs a multi-granularity adaptive semantic segmentation algorithm to parse the document set to be processed and generate semantic data blocks, including: Multi-granularity scanning is performed on the documents to be processed in the document set to identify candidate segmentation points at different granularity levels and obtain a sequence of candidate segmentation points. Calculate the topic consistency score of the text units on both sides of each candidate segmentation point, and select the optimal segmentation point sequence from the candidate segmentation point sequence based on the topic consistency score and a preset adaptive threshold. The adaptive threshold is dynamically calculated based on at least one of document type, text length and domain keyword density. Based on the optimal segmentation point sequence, semantic segmentation is performed on the document to be processed, and a hierarchical relationship between the segmentation blocks is constructed according to the granular nesting relationship in the semantic segmentation process, to obtain semantic data blocks with hierarchical relationships.

3. The task-based control method driven by large language model pre-analysis and preprocessing as described in claim 2, characterized in that, The dynamic calculation method includes: An adaptive threshold calculation formula is obtained by fitting the document type, text length, domain keyword density, and topic consistency score of manually annotated segmented documents with different granularity segmentation points. Based on the document type, paragraph length, and domain keyword density corresponding to the document to be processed, the adaptive threshold corresponding to the document to be processed is obtained by using the adaptive threshold calculation formula.

4. The task-based control method driven by large language model pre-analysis and preprocessing as described in claim 1, characterized in that, The pre-built feature extraction model is used to extract features from each semantic data block to obtain a multi-dimensional feature set, including: A pre-built feature extraction model is used to process each semantic data block to obtain the topic vector, task adaptation type label, and cross-block correlation matrix of the semantic data block; the feature extraction model includes a semantic encoder and a multi-task output head; A pre-built risk assessment engine is used to perform data conflict detection on the topic vector to obtain the data conflict risk level of the semantic data block; Based on the topic vector, the task adaptation type label, the cross-block correlation matrix, and the data conflict risk level, a multi-dimensional feature set of the semantic data block is constructed.

5. The task-based control method driven by large language model pre-analysis and preprocessing as described in claim 4, characterized in that, The structured index table has a hierarchical, cascaded architecture and includes at least: The task adaptation sub-index layer is constructed based on the index identifier and the task adaptation type label corresponding to each semantic data block, and is used to realize fast filtering based on task type. The semantic adaptation sub-index layer is constructed based on the index identifier and topic vector corresponding to each semantic data block, and is used to realize topic similarity-based retrieval. The context-related sub-index layer is constructed based on the index identifiers corresponding to each semantic data block and the cross-block correlation matrix, and is used to support the retrieval of related contexts.

6. The task-based control method driven by large language model pre-analysis and preprocessing as described in claim 5, characterized in that, The step of dynamically retrieving a candidate data block list from the structured index table based on the received user query task includes: The received user query task is parsed to obtain the query task type, semantic query vector, and context association requirements; Based on the query task type, a search is performed in the task adaptation sub-index layer to obtain the first candidate set; Based on the semantic query vector, a similarity search is performed on the first candidate set in the semantic adaptation sub-index layer to obtain the second candidate set; Based on the aforementioned context association requirements, the elements in the second candidate set are associated and expanded in the context association sub-index layer to obtain a third candidate set. A list of candidate data blocks is obtained based on the semantic data blocks associated with the second candidate set and the semantic data blocks associated with the third candidate set.

7. The task-based control method driven by large language model pre-analysis and preprocessing as described in claim 6, characterized in that, The step of generating a data block loading strategy based on the multidimensional feature set associated with each candidate data block in the candidate data block list includes: Based on the multidimensional feature set associated with each candidate data block in the candidate data block list, the task fit and topic relevance between each multidimensional feature set and the user query task are obtained; based on the data conflict risk level associated with the candidate data block, the risk penalty coefficient is obtained. A comprehensive priority score is obtained based on the task suitability, the topic relevance, and the risk penalty coefficient. Based on the hierarchical relationship between each index identifier, an initial loading queue is obtained, and based on the comprehensive priority score, the candidate data blocks at the same level of the initial loading queue are adjusted to obtain a data block loading strategy.

8. The task-based control method driven by large language model pre-analysis and preprocessing as described in claim 7, characterized in that, The step of processing the candidate data blocks by the selected quantized large language model according to the data block loading strategy, and generating intermediate processing results corresponding to each candidate data block, includes: Based on the context window length limit of the selected quantized large language model, set a fixed capacity for the sliding window controller; According to the data block loading strategy, the contents of the corresponding candidate data blocks are filled into the current sliding window of the sliding window controller, and a data block length check is performed during the content filling process to ensure that the total length of the current sliding window is less than or equal to the fixed capacity. Perform a secondary semantic split on a single candidate data block whose length exceeds the fixed capacity, and assign an extended index identifier to the split sub-semantic data blocks; The content of the current sliding window is input into the quantized large language model to generate an intermediate processing result associated with the index identifier or the extended index identifier.

9. The task-based control method driven by large language model pre-analysis and preprocessing as described in claim 4, characterized in that, The step of integrating the intermediate processing results according to the corresponding index identifier to obtain the task processing result includes: Based on the hierarchical relationship of the index identifier or the extended index identifier associated with the intermediate processing results, the intermediate processing results are integrated to obtain the initial task processing results; Based on the data conflict risk level of the semantic data block associated with the intermediate processing result, the initial task processing result is semantically conflict corrected to obtain the task processing result.

10. A task-based control system driven by large language model pre-analysis and preprocessing, used to implement the task-based control method driven by large language model pre-analysis and preprocessing as described in any one of claims 1-9, characterized in that, The system includes: a document parsing module, a feature extraction module, a data block retrieval module, a loading strategy generation module, and a task processing module; The document parsing module is used to parse the document set to be processed using a multi-granularity adaptive semantic segmentation algorithm, generate semantic data blocks, and assign an index identifier with a hierarchical relationship to each semantic data block. The feature extraction module is used to extract features from each of the semantic data blocks using a pre-built feature extraction model to obtain a multi-dimensional feature set. The data block retrieval module is used to construct a structured index table based on the semantic data block, the index identifier, and the multidimensional feature set, and dynamically retrieve a candidate data block list from the structured index table according to the received user query task. The loading strategy generation module is used to generate a data block loading strategy based on the multidimensional feature set associated with each candidate data block in the candidate data block list. The data block loading strategy is set to be determined based on the task adaptability and topic relevance between the multidimensional feature set and the user query task, as well as the hierarchical relationship between the index identifiers. The task processing module is used to control the selected quantized large language model to process the candidate data blocks according to the data block loading strategy, generate intermediate processing results corresponding to each candidate data block, and integrate the intermediate processing results according to the corresponding index identifier to obtain the task processing result.

Images