A batch text processing method, device, equipment and medium

Abstract

The application belongs to the technical field of file processing, and relates to a batch text processing method, device, equipment and medium. The method comprises the following steps: obtaining batch texts to obtain a text length coefficient of a file; mapping each sentence of the file into a vector to predict a sentence-level dependency arc, obtain root label diversity and average dependency depth, and further obtain a structure complexity coefficient of the file; obtaining a professional term density coefficient of the file according to a total number of text characters of the file; obtaining a comprehensive complexity coefficient of each file according to the text length coefficient, the structure complexity coefficient and the professional term density coefficient to obtain a plurality of file groups; generating a corresponding group report for each file group; splicing all group reports to obtain a total report; and performing error analysis on the total report to obtain a final report. The application can directly generate a processing result of batch texts.

Description

Technical Field

[0001] This application relates to the field of document processing technology, and in particular to a method, apparatus, device, and medium for processing batch text. Background Technology

[0002] With the development of the times and the advancement of technology, the number of various documents is growing exponentially. Especially in the wave of digital transformation, enterprises' core assets are expanding at an unprecedented rate from structured data to unstructured files, including documents, images, videos, code, and logs. These massive amounts of files carry the enterprise's critical knowledge, business processes, and compliance credentials, serving as the lifeline for organizational operations and innovation. Correspondingly, the demand for batch text processing is also increasing daily.

[0003] In existing technologies, batch text processing is generally done manually, requiring staff to process each document one by one.

[0004] However, the above approach has the following problems:

[0005] 1) The accuracy of manual review of batch texts is insufficient.

[0006] 2) Manual review of batch texts is inefficient and slow.

[0007] 3) Manual review of batch texts is costly.

[0008] 4) The governance report is generated directly from all files using a large artificial intelligence model. However, due to the limited context length, it cannot be generated by splicing together the files. Summary of the Invention

[0009] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for batch text processing to address the aforementioned technical problems. This method can directly generate batch text processing results with high accuracy, high efficiency, fast speed, and low cost.

[0010] A method for processing batch text includes:

[0011] Get a batch of text and calculate the text length coefficient of the file based on the total number of text characters in the file;

[0012] Each sentence in the file is mapped to a vector to obtain the file's context vector. Based on the file's context vector, sentence-level dependency arcs are predicted, the root label distribution of the file is output, and the number of root label types in the file is obtained to obtain the root label diversity. According to the sentence-level dependency arcs, the dependency matrix is ​​obtained, and the maximum dependency depth is solved sentence by sentence to obtain the average dependency depth. According to the root label diversity and the average dependency depth, the structural complexity coefficient of the file is obtained.

[0013] Based on the total number of text characters in the file, the frequency of occurrence of technical terms is obtained, and combined with the number of sentences in the file, the technical term density coefficient of the file is obtained.

[0014] The overall complexity coefficient of each file is obtained based on the text length coefficient, structural complexity coefficient, and terminology density coefficient; based on the overall complexity coefficient, the batch of texts is grouped to obtain multiple file groups;

[0015] For each file group, generate a corresponding group report; combine all group reports to obtain a master report; perform error analysis on the master report to obtain the final report for batch text processing.

[0016] In one embodiment, each sentence of the file is mapped to a vector to obtain the file's context vector, including:

[0017] Using regular expressions, the file is segmented into multiple sentences based on the punctuation at the end of each sentence;

[0018] Each word in the sentence is mapped to a vector using a 128-dimensional FastText to obtain a word vector;

[0019] All word vectors of each sentence are concatenated into a sequence according to word order, input into a two-layer bidirectional long short-term memory network, and then input into a four-head multi-head attention mechanism to obtain the context representation of each sentence;

[0020] The context vector of the file is obtained based on all context representations.

[0021] In one embodiment, based on the file's context vector, sentence-level dependency arcs are predicted, the root label distribution of the file is output, and the number of root label types in the file is obtained to determine root label diversity, including:

[0022] Based on the context vector of the file, a dual affine attention decoder is used to predict sentence-level dependency arcs and output the root label distribution of the file using a normalized exponential function.

[0023] Based on the root tag distribution of the file, perform a 64-bit rolling hash on the root tag of each sentence and perform atomic operations on the GPU to remove duplicates, thereby obtaining the number of root tag types in the file;

[0024] The root tag diversity is obtained based on the number of root tag types in the file.

[0025] In one embodiment, the dependency matrix is ​​obtained based on sentence-level dependency arcs, and the maximum dependency depth is solved sentence by sentence to obtain the average dependency depth, including:

[0026] Based on the sentence-level dependency arcs, the dependency matrix is ​​obtained;

[0027] The dependency matrix is ​​compressed in CSR format and then input into the CUDA kernel function to perform parallel BFS, and the maximum dependency depth is solved line by line.

[0028] The average dependency depth is obtained based on the maximum dependency depth.

[0029] In one embodiment, the batch of text is grouped according to the comprehensive complexity coefficient to obtain multiple file groups, including:

[0030] Based on the comprehensive complexity coefficient, the batch of text is grouped and constrained to obtain the number of files in different groups;

[0031] Based on the number of files in different groups, set the expected proportion weight vector for each group; based on the comprehensive complexity coefficient and the expected proportion weight vector for each group, group the batch of text to obtain multiple initial groups; calculate the processing difficulty within each initial group and the splicing error rate within each initial group to obtain the objective function for the current grouping.

[0032] The stochastic gradient descent algorithm or grid search algorithm is used to update the overall complexity coefficient and the expected proportion weight vector of each group, and then regroup and recalculate the objective function of the current group. This process is repeated until the preset conditions are met, at which point the updating and iteration stop, resulting in multiple parameter sets. Each parameter set includes the overall complexity coefficient, the expected proportion weight vector of each group, and the objective function.

[0033] Based on the parameter group containing the largest objective function, the optimal comprehensive complexity coefficient and the expected proportion weight vector of each group are obtained to group the batch of text and obtain multiple file groups.

[0034] In one embodiment, a corresponding group report is generated for each file group, including:

[0035] For each filegroup, sort them according to the timestamp of each file to obtain the initial sequence;

[0036] Calculate the similarity between any two files, and arrange the files whose similarity meets the preset conditions adjacently to optimize the initial sequence and obtain the optimized sequence;

[0037] The text of all files is concatenated according to the optimized sequence, and markers are inserted at the concatenation positions to obtain the concatenated text.

[0038] Extract the summary from the concatenated text to obtain a summary of the concatenated text;

[0039] Perform entity recognition on the concatenated text to generate a list of entities in the concatenated text;

[0040] Keyword recognition is performed on the concatenated files to obtain the keyword set of the concatenated text;

[0041] Error detection and verification are performed on the concatenated file to obtain a set of errors in the concatenated text;

[0042] The summary, entity list, keyword set, and error set of the concatenated text are used to generate a report based on the index, resulting in a group report for each file group.

[0043] In one embodiment, calculating the similarity between any two files includes:

[0044] Obtain multiple sample files and get the semantic sentence vector for each file; randomly select samples and train PCA to obtain the projection matrix and feature values;

[0045] For any semantic sentence vector, calculate the dimension-reduced vector based on the projection matrix; then, based on the eigenvalues, recalibrate the dimension-reduced vector dimension by dimension to obtain the recalibrated vector.

[0046] Design weights, and calculate weighted cosine similarity based on the weights and the recalibrated vectors; perform a linear mapping on the weighted cosine similarity to obtain the similarity between the two files.

[0047] A batch text processing apparatus, comprising:

[0048] The first module is used to obtain a batch of text and to obtain the text length coefficient of the file based on the total number of text characters in the file;

[0049] The second module maps each sentence of the file to a vector to obtain the file's context vector; based on the file's context vector, it predicts sentence-level dependency arcs, outputs the root label distribution of the file, and obtains the number of root label types to obtain root label diversity; based on the sentence-level dependency arcs, it obtains the dependency matrix and solves for the maximum dependency depth sentence by sentence to obtain the average dependency depth; based on the root label diversity and the average dependency depth, it obtains the file's structural complexity coefficient.

[0050] The third module is used to obtain the number of times technical terms appear based on the total number of text characters in the file, and to obtain the technical term density coefficient of the file by combining it with the number of sentences in the file.

[0051] The fourth module is used to obtain the comprehensive complexity coefficient of each file based on the text length coefficient, structural complexity coefficient, and technical terminology density coefficient; and to group the batch of texts according to the comprehensive complexity coefficient to obtain multiple file groups.

[0052] The fifth module is used to generate a corresponding group report for each file group; to combine all group reports to obtain a master report; and to perform error analysis on the master report to obtain the final report for batch text processing.

[0053] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described method.

[0054] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0055] The aforementioned batch text processing method, apparatus, computer equipment, and storage medium, through hierarchical aggregation and dynamic programming combined with large-scale model text analysis functions, split large-scale files into batches, achieving automated and efficient batch text processing. This not only significantly improves processing efficiency and reduces data volume, but also ensures the integrity and accuracy of information by progressively extracting key information, solving the contextual constraint problem of existing large models and greatly improving the quality of batch text processing. Simultaneously, dynamic programming is used to optimize clustering and generate reports, and various error scenarios are statistically analyzed in detail during file processing, further improving the accuracy and efficiency of batch text processing and providing a strong basis for subsequent optimization and improvement. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating a batch text processing method in one embodiment;

[0057] Figure 2 This is a structural block diagram of a batch text processing device in one embodiment;

[0058] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0060] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. In the description of this application, "multiple sets" means at least two sets, such as two sets, three sets, etc., unless otherwise explicitly specified.

[0061] In this application, unless otherwise expressly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two elements or the interaction between two elements, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0062] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0063] This application provides a method for batch text processing, such as... Figure 1 The flowchart shown, in one embodiment, includes:

[0064] Step 101: Obtain a batch of text and calculate the text length coefficient of the file based on the total number of text characters in the file.

[0065] Specifically:

[0066] Get batch text;

[0067] The file's text length coefficient is obtained based on the total number of text characters in the file.

[0068] ;

[0069] In the formula, This is the text length coefficient; It is a natural constant; The steepness coefficient represents the number of characters that deviate from the center point (i.e., One unit, how many times is the exponent magnified? Total number of text characters; This is the "watershed" for the Sigmoid function, that is, the input value corresponding to the function's output value of 0.5. For example, when the number of characters is equal to 3000, the function output value is 0.5; when the number of characters is less than 3000, the function output value moves closer to 0; and when the number of characters is greater than 3000, the function output value moves closer to 1.

[0070] In this step, each file has a corresponding text length coefficient. The text length coefficient measures the size of the file's text content. The length of typical sample files is concentrated between 1000 and 5000 characters. A coefficient between 0 and 1 is obtained by using the Sigmoid function for non-linear mapping. This not only eliminates the need for additional normalization steps and reduces the risk of dimensionality explosion in subsequent model inputs, but also makes the weight differences between sample files of different lengths more significant and the gradient updates more stable, thereby improving the accuracy and efficiency of batch text processing.

[0071] It should be noted that how to obtain batch text, the Sigmoid function, and the "watershed" of the Sigmoid function are all existing technologies and will not be elaborated here.

[0072] Step 102: Map each sentence of the file to a vector to obtain the file's context vector; based on the file's context vector, predict sentence-level dependency arcs, output the root label distribution of the file, and obtain the number of root label types to obtain root label diversity; based on the sentence-level dependency arcs, obtain the dependency matrix, and solve for the maximum dependency depth sentence by sentence to obtain the average dependency depth; based on the root label diversity and average dependency depth, obtain the file's structural complexity coefficient.

[0073] Specifically:

[0074] Using regular expressions, the file is segmented into multiple sentences based on sentence-end punctuation. Each sentence's words are mapped to vectors using a 128-dimensional FastText algorithm, resulting in word vectors. All word vectors from each sentence are concatenated into a sequence according to word order and input into a two-layer bidirectional long short-term memory network (bi-LSTM), followed by a 4-head multi-head attention mechanism to obtain the context representation of each sentence. Based on all context representations, the file's context vector is obtained.

[0075] Based on the file's context vector, a biaffine attention decoder is used to predict sentence-level dependency arcs, and the root label distribution of the file is output using a normalized exponential function (softmax function). According to the root label distribution of the file, a 64-bit rolling hash is performed on the root label of each sentence, and atomic operations are performed on the GPU to remove duplicates, thus obtaining the number of root label types in the file. The root label diversity is obtained based on the number of root label types in the file.

[0076] The dependency matrix is ​​obtained based on the sentence-level dependency arcs; the dependency matrix is ​​compressed in CSR format and then input into the CUDA kernel function to perform parallel BFS, and the maximum dependency depth is solved sentence by sentence; the average dependency depth is obtained based on the maximum dependency depth.

[0077] The structural complexity coefficient of the file is obtained based on the root tag diversity and average dependency depth.

[0078] More specifically:

[0079] Using regular expressions, the file is segmented into multiple sentences based on sentence-ending punctuation. Each sentence's words are then mapped to vectors using a 128-dimensional FastText algorithm, resulting in word vectors.

[0080]

[0081] In the formula, Word vectors;

[0082] The word vectors of each sentence are concatenated into a sequence according to word order and input into a two-layer bidirectional long short-term memory network (bi-LSTM). The hidden layer dimension of the Bi-LSTM is 256. Then, a 4-head multi-head attention mechanism is input to obtain the context representation of each sentence.

[0083]

[0084] In the formula, The temporal hidden state matrix output by a two-layer bidirectional long short-term memory network (bi-LSTM); For 4 heads multi-head attention The context representation of the sentence obtained after weighted summation is the sentence-level context vector.

[0085] The context vector of the file is obtained by combining all the context representations.

[0086] Based on the file's context vector, a biaffine attention decoder is used to predict sentence-level dependency arcs, and the root label distribution of the file is output using a normalized exponential function (softmax function). According to the root label distribution, a 64-bit rolling hash is performed on the root label of each sentence, and atomic operations are executed on the GPU to remove duplicates, obtaining the number of root label types in the file. Based on the number of root label types, the root label diversity is obtained.

[0087]

[0088] In the formula, For root tag diversity; This represents the number of root tag types. The number of sentences in the file;

[0089] Based on sentence-level dependency arcs, the dependency matrix is ​​obtained; the dependency matrix is ​​compressed in CSR format and input into the CUDA kernel function to perform parallel BFS, and the maximum dependency depth is calculated sentence by sentence; based on the maximum dependency depth, the average dependency depth is obtained.

[0090]

[0091] In the formula, This represents the average dependency depth. To obtain the average; Maximum dependency depth;

[0092] Based on root tag diversity and average dependency depth, the structural complexity coefficient of the file is obtained:

[0093] ;

[0094] ;

[0095] ;

[0096] In the formula, is the structural complexity coefficient.

[0097] In this step, each sentence in the file is mapped to a vector, and sentence-level dependency arcs are predicted to obtain root label diversity and average dependency depth, thereby obtaining the structural complexity coefficient of the file. This can accurately quantify syntactic depth and sentence diversity, significantly improve the distinguishability of long and difficult sentences and achieve zero dependence on external weights, further improving the accuracy and efficiency of batch text processing.

[0098] It should be noted that regular expressions, how to segment files into sentences, FastText, two-layer bidirectional long short-term memory networks (two-layer Bi-LSTM), 4-head multi-head attention mechanisms, biaffine attention decoders, how to predict sentence-level dependency arcs, normalized exponential functions (softmax functions), 64-bit rolling hashing, atomic deduplication, how to obtain dependency matrices, how to compress in CSR format, CUDA kernel functions, how to perform parallel BFS, and how to solve for maximum dependency depth sentence by sentence are all existing technologies and will not be elaborated here.

[0099] Step 103: Based on the total number of text characters in the file, obtain the number of times technical terms appear, and combine this with the number of sentences in the file to obtain the technical term density coefficient of the file.

[0100] Specifically:

[0101] The number of times technical terms appear is obtained based on the total number of text characters in the file;

[0102] Based on the frequency of occurrence of technical terms and the number of sentences in the document, the technical term density coefficient of the document is obtained:

[0103] ;

[0104] in, Density coefficient is a technical term. The number of times technical terms appear. The number of sentences in the file.

[0105] In this step, technical terms are the standardized names for specific things used in a particular field. Determining whether a character belongs to a technical term and how to obtain the frequency of occurrence of a technical term are all existing technologies and will not be elaborated here.

[0106] Step 104: Obtain the comprehensive complexity coefficient of each file based on the text length coefficient, structural complexity coefficient, and technical terminology density coefficient; group the batch of texts according to the comprehensive complexity coefficient to obtain multiple file groups.

[0107] Specifically:

[0108] The overall complexity coefficient of each file is obtained based on the text length coefficient, structural complexity coefficient, and technical terminology density coefficient.

[0109] Based on the comprehensive complexity coefficient, the batch of text is grouped and constrained to obtain the number of files in different groups;

[0110] The batch of text is grouped according to the number of files in different groups to obtain multiple file groups.

[0111] More specifically:

[0112] Based on the text length coefficient, structural complexity coefficient, and terminology density coefficient, the overall complexity coefficient of each file is obtained:

[0113]

[0114] In the formula, This is the overall complexity coefficient; This is the text length coefficient; The structural complexity coefficient; Density coefficient is a technical term.

[0115] Based on the overall complexity coefficient, the batch of text is grouped to obtain the number of files in different groups:

[0116] ;

[0117] In the formula, The number of files in different groups; To assign the value of the right-hand expression to the left-hand expression repeatedly. ;

[0118] Based on the number of files in different groups, set the expected proportion weight vector for each group. Then, based on the overall complexity coefficient and the expected proportion weight vector for each group, group the batch of text into multiple initial groups. Next, update the overall complexity coefficient and the expected proportion weight vector for each group to obtain the optimal overall complexity coefficient and the optimal expected proportion weight vector for each group. Finally, regroup the batch of text to obtain multiple file groups (specifically: based on the number of files in different groups). Set the expected proportion weight vector for each group. Based on the comprehensive complexity coefficient and the expected proportion weight vector of each group The batch of text is grouped to obtain multiple initial groups; according to Calculate the intra-group processing difficulty for each initial group, based on Calculate the intra-group splicing error rate for each initial group, and based on... Calculate the objective function for the current grouping (where, Indicates the difficulty of processing within the group. This represents the average of the text length coefficients for all files within the group. This represents the average of the structural complexity coefficients of all files within the group. This represents the average of the technical terminology density coefficients across all documents within the group. This indicates the splicing error rate within the group. This indicates the number of instances where cross-sentence dependency arc connections within the group failed. This indicates the number of items with inconsistent dependency tags within the group. Indicates the total number of sentences in the group. (This represents the penalty coefficient); the overall complexity coefficient is calculated using either stochastic gradient descent or grid search algorithms. and the expected proportion weight vector of each group Update the data, regroup the data, and recalculate the objective function for each group. The process is repeated iteratively until the preset condition (two consecutive rounds) is met. When the increase is less than 0.1%, stop updating and iterating, resulting in multiple parameter sets, each containing one parameter. , , Choose the largest one. In the parameter group and The optimal and According to the optimal and (Batch text is grouped to obtain multiple file groups).

[0119] In this step, dynamic grouping is performed based on the overall complexity coefficient, which can re-cluster the files according to their overall complexity, making the difficulty within the group balanced and minimizing splicing errors. It can also adaptively rearrange the data when it changes, continuously improving the efficiency and accuracy of subsequent processing without manual intervention.

[0120] It needs to be explained how to set the expected proportion weight vector for each group, and how to determine the overall complexity coefficient. and the expected proportion weight vector of each group Grouping batches of text, how to obtain the number of failed cross-sentence dependency arc connections within a group, how to obtain the number of inconsistent dependency labels within a group, how to set the penalty coefficient, stochastic gradient descent algorithm, and grid search algorithm are all existing technologies and will not be elaborated here.

[0121] Step 105: Generate a corresponding group report for each file group; combine all group reports to obtain a master report; perform error analysis on the master report to obtain the final report for batch text processing.

[0122] Specifically:

[0123] For each file group, sort the files according to their timestamps to obtain an initial sequence; calculate the similarity between any two files and arrange files with similarity meeting preset conditions adjacently to optimize the initial sequence, obtaining an optimized sequence; concatenate the main text of all files according to the optimized sequence and insert placeholders at the concatenation positions to obtain concatenated text; extract a summary from the concatenated text to obtain a summary of the concatenated text; perform entity recognition on the concatenated text to generate an entity list of the concatenated text; perform keyword recognition on the concatenated text to obtain a keyword set of the concatenated text; perform error detection on the concatenated text and review it to obtain an error set of the concatenated text; generate a report based on the summary, entity list, keyword set, and error set of the concatenated text according to the index to obtain a group report for each file group;

[0124] Starting with the first group report, all group reports preceding the current group report are concatenated to obtain a concatenated report; the base length, complexity coefficient, and length coefficient of the concatenated report are calculated to obtain the summary length of the concatenated report; based on the summary length of the concatenated report, a large language model is used for processing to obtain a batch of text reports.

[0125] Error analysis is performed on the master report to obtain the final report of batch text processing.

[0126] More specifically:

[0127] For each filegroup, sort them according to the timestamp of each file to obtain the initial sequence. Using a 12-layer, 768-dimensional Transformer encoder, the cosine similarity between any two files is calculated.

[0128]

[0129] In the formula, The cosine similarity between the two files; This represents the number of files in the filegroup.

[0130] A greedy insertion algorithm is used to arrange files whose cosine similarity meets a preset condition adjacently to optimize the initial sequence. Optimization is performed to obtain the optimized sequence. According to the optimized sequence Concatenate the main text of all files and insert placeholders at the concatenation positions (the placeholders can be: ###g{f}###, where g represents the group number and f represents the file number within the group; g and f can be obtained together when multiple file groups are obtained) to obtain a position mapping table (the position mapping table is a series of position pairs composed of g and f, which can provide anchor points for subsequent fragment-level positioning), and at the same time obtain the concatenated text;

[0131] The concatenated text is fed into the same Transformer using a 512-token sliding window. Sentences with a probability p > 0.7 are retained and compressed to 8% of the original text length for summary extraction, resulting in a summary of the concatenated text. A 24-layer, 1024-dimensional BioBERT-large layer is used for BERT-NER to identify four types of entities: terms, amounts, time limits, and entities, generating an entity list for the concatenated text. TF-IDF is calculated on the concatenated file, and the Top-K (K=20) high-weight words are selected for keyword identification, resulting in a keyword set for the concatenated text. A millisecond-level lightweight rule engine is used to quickly identify candidate errors such as missing title format, incorrect amount precision, conflicting validity levels, expired time limits, and classification mismatches using five regular expressions r1–r5, performing error detection on the concatenated file and outputting a candidate set C_rule={(type, pos, (where type represents the error category of the candidate, pos represents the starting index of the erroneous segment in the full-text character sequence, and raw represents the raw string captured by the regular expression on the spot), and each candidate and its ±128 token context are input into the large language model for verification, retaining only the triples R= with a confidence score ≥0.85.<type, exact_span, fix> (Where, exact_span represents the "start-end" index for precisely locating the erroneous segment in the full-text character sequence, and fix represents the directly applicable repair text provided by the large language model.) To avoid redundant judgments, the error set of the concatenated text is obtained. The summary, entity list, keyword set, and error set of the concatenated text are combined with the position mapping table and written to JSON according to the index "group_id→file_id→(start_char, end_char)" (where group_id represents the batch ID, file_id represents the file ID, (start_char, end_char) represents the character-level coordinates, and "group_id→file_id→(start_char, end_char)" represents a three-level coordinate system of "first find the batch → then find the file → finally accurate to the character position" to ensure that the program can lock the original text segment). The same large oracle model renders and generates a report, obtaining the group report for each file group. (Note: How to obtain type, pos, raw, exact_span, fix, group_id, file_id, start_char, and end_char are all existing technologies and will not be elaborated here.)

[0132] Starting with the first group report, concatenate all group reports preceding the current group report to obtain a concatenated report dp[i] (dp[i] represents the first i+1 reports; for example, i∈[0,999], when i=0, dp[0] represents the first 1 report, and when i=999, dp

[999] represents the first 1000 reports); calculate the base length of the concatenated report:

[0133]

[0134] In the formula, This serves as the base length for compiling the report. The total number of characters in the assembled report;

[0135] Calculate the complexity coefficient of the splicing report:

[0136]

[0137] In the formula, This represents the complexity coefficient for assembling the report.

[0138] Calculate the length factor of the splicing report:

[0139]

[0140] In the formula, This is the length factor for the spliced ​​report;

[0141] Based on the base length, complexity coefficient, and length coefficient of the assembled report, the summary length of the assembled report is obtained:

[0142]

[0143] In the formula, This is the length of the summary of the assembled report;

[0144] Based on the length of the summary of the assembled report, a large language model (such as LLM_extract) is used for processing to obtain a total report of the batch text;

[0145] A rule engine is used to perform error analysis on the overall report, and a large language model is used for review, outputting a triplet of error features (misalignment features include: error type, location, and correction) (i.e., R=<type, exact_span, fix> This will generate the final report for batch text processing.

[0146] In this step, the optimized sequence can balance temporal order and semantic coherence, improving the accuracy of the generated report. When generating group reports, a dynamic programming method is used for step-by-step optimization and aggregation, so that long and complex texts automatically obtain more summary space and short and concise texts avoid redundancy. The dynamic length strategy for generating reports is superior to the fixed length strategy in multiple text metrics such as compression rate, readability and information completeness, which can improve the quality of the generated report, and the extracted summary can better cover the core information of the text.

[0147] It should be noted that the following aspects are existing technologies and will not be elaborated upon here: how to obtain the file timestamp, the Transformer encoder, how to use the Transformer encoder to calculate cosine similarity, the greedy insertion method, preset conditions, the 512-token sliding window, how to perform summary extraction, BioBERT-large, how to perform BERT-NER, how to calculate TF-IDF, how to extract the Top-K (K=20) high-weight words, the millisecond-level lightweight rule engine, five regular expressions, the large language model, how to write the index to JSON, how to render and generate reports, the rule engine, and how to use the rule engine for error analysis.

[0148] In this embodiment, preferably, step 105, calculating the similarity between any two files, may include:

[0149] Multiple sample files were obtained, and each file was segmented using the Jieba word segmentation library. The segments were then input into the nomic-embed-text model to obtain the semantic sentence vector for each file.

[0150]

[0151] In the formula, The semantic sentence vector for each file; It is the set of real numbers; This refers to the number of dimensions; in this case, it is 768 dimensions.

[0152] In the open-source Small Chinese Corpus, m=1e5 samples are randomly selected and PCA is trained to obtain the projection matrix and eigenvalues:

[0153] ;

[0154] ;

[0155] In the formula, The projection matrix; It is the set of real numbers; This represents the number of dimensions after dimensionality reduction, which is 512 dimensions in this case. The number of dimensions; The first output of PCA i The eigenvalue, which is the eigenvalue... i The variance of each principal component;

[0156] For any semantic sentence vector, calculate the dimension-reduced vector based on the projection matrix:

[0157] ;

[0158] ;

[0159] In the formula, The vector after dimensionality reduction; The projection matrix; For semantic sentence vectors; This represents the number of dimensions after dimensionality reduction, which is 512 dimensions in this case.

[0160] Based on the eigenvalues, the dimension-reduced vector is recalibrated dimension by dimension to obtain the recalibrated vector:

[0161] ;

[0162] ;

[0163] In the formula, For the first i Each recalibrated vector; For the first i A vector after dimensionality reduction; The first output of PCA i One eigenvalue; For coefficients;

[0164] Design weights, and calculate weighted cosine similarity based on the weights and the recalibrated vectors:

[0165] ;

[0166] ;

[0167] In the formula, For the first i Dimensional softmax weights; The first output of PCA j The eigenvalue, which is the eigenvalue... j The variance of each principal component; for and Weighted cosine similarity; For the first comparison document i Each recalibrated vector;

[0168] The weighted cosine similarity is linearly mapped to 0–100 to obtain the semantic similarity score:

[0169]

[0170] In the formula, This refers to the semantic similarity score;

[0171] The semantic similarity score is used as the similarity between the two files.

[0172] Compared to existing methods that use direct cosine similarity calculation, the above settings reduce redundant dimension weights to near zero by using variance recalibration and softmax weighting, while amplifying the weights of core semantic dimensions. This results in S≈99.8 for synonymous sentences, S≈87.4 for near-synonymous sentences, and S≈12.6 for irrelevant sentences, significantly increasing the distinguishability and reducing the differences in long text length and high-frequency function word noise. This increases the accuracy of text similarity and improves the accuracy and efficiency of batch text processing.

[0173] It should be noted that the Jieba word segmentation library, nomic-embed-text model, open-source Small Chinese Corpus, PCA training, and linear mapping are all existing technologies and will not be elaborated upon here.

[0174] The aforementioned batch text processing method combines hierarchical aggregation and dynamic programming with large-scale model text analysis capabilities to split large-scale files into batches, achieving automated and efficient batch text processing. This not only significantly improves processing efficiency and reduces data volume but also ensures the integrity and accuracy of information by progressively extracting key information, thus solving the contextual constraint problem of existing large models and greatly improving the quality of batch text processing. Furthermore, dynamic programming is used to optimize clustering and generate reports, and various error scenarios are statistically analyzed in detail during file processing, further improving the accuracy and efficiency of batch text processing and providing a strong basis for subsequent optimization and improvement.

[0175] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0176] In one specific embodiment, a batch text set is obtained. There are a total of 1000 articles, and the character length L of each article is uniformly distributed in the interval [1000, 5000].

[0177] Calculate the text length coefficient for each file:

[0178]

[0179] ...

[0181]

[0182] Its min=0.100, max=0.500, mean=0.300, and standard deviation=0.115;

[0183] Calculate the root tag diversity for each document:

[0184]

[0185] ...

[0187]

[0188] Its min=0.280, max=0.667, mean=0.450, and standard deviation=0.089;

[0189] Calculate the structural complexity coefficient of each file:

[0190]

[0191] ...

[0193]

[0194] Its min= 0.007, max=0.033, mean=0.013, standard deviation=0.008;

[0195] Calculate the terminology density coefficient for each document:

[0196]

[0197] ...

[0199]

[0200] Its min=2.800, max=6.667, mean=5.000, standard deviation=0.816;

[0201] Then, the overall complexity coefficient is calculated based on the coefficients obtained above, and the batch of text is grouped to obtain multiple file groups; for each file group, a corresponding group report is generated; all group reports are concatenated to obtain a total report; error analysis is performed on the total report to obtain the final report of batch text processing;

[0202] Finally, a total of 867 errors in 5 categories were detected, including 248 format errors, 173 punctuation errors, 171 consistency errors, 152 integrity errors, and 123 timeliness errors. Based on the error type, unified correction opinions were generated: format errors were automatically filled with spaces or table borders, punctuation errors were standardized with Chinese symbols, consistency errors were automatically replaced with conflicting terms, integrity errors were filled with missing dates or clause numbers, and timeliness errors were prompted to update obsolete standards.

[0203] In another specific embodiment: the objects to be processed are 10 corporate management system documents, with a total character count of approximately 50,000 characters, a single document length of 3,000-8,000 characters, and document types such as personnel management system, training management method, performance appraisal system, financial management system, and quality management system.

[0204] 1. First stage: Generate group report (taking a specific file group as an example)

[0205] Items to be processed: Personnel management system + Training management regulations + Performance appraisal system

[0206] Summary generation: 320 characters

[0207] Entity extraction: 23 (including organizational structure, job roles, policy documents, management activities, etc.)

[0208] Keyword extraction: 15

[0209] Errors detected: 3 (inconsistent formatting, inconsistent terminology, missing cross-references)

[0210] 2. Second stage: Generate the splicing report (cumulative from three file groups)

[0211] Basic length: 1847 characters (combined from the first three digests)

[0212] Complexity coefficient: 1.35 (calculated based on the association density of 54 different entities)

[0213] Length factor: 0.82 (information redundancy 18%)

[0214] Abstract length calculation: 1847 × 1.35 × 0.82 = 2043 characters

[0215] 3. Third stage: Generating the final report

[0216] Input to the large language model: a 2043-character summary

[0217] Output of the large language model: 1338 characters after compression

[0218] Compression rate: 74.3%

[0219] 4. Fourth stage: Generating the final report

[0220] Error analysis: 11 errors (4 instances of duplicate entities, 2 instances of logical conflicts, 3 instances of data mismatch, and 2 instances of inconsistent formatting).

[0221] Reviewed report: 1285 characters

[0222] Information extraction accuracy: 96.8%

[0223] Entity recognition accuracy: 94.2% (51 out of 54 entities were accurately identified)

[0224] Semantic consistency score: 91.7% (88.6% before correction)

[0225] Processing speed: 15.7 blocks / second

[0226] Context utilization: 85.4% (processed within a 4000-character limit).

[0227] This application also provides a batch text processing apparatus, such as... Figure 2 As shown, in one embodiment, it includes: a first module 201, a second module 202, a third module 203, a fourth module 204, and a fifth module 205, wherein:

[0228] The first module 201 is used to obtain a batch of text and to obtain the text length coefficient of the file based on the total number of text characters in the file;

[0229] The second module 202 is used to map each sentence of the file into a vector to obtain the context vector of the file; based on the context vector of the file, it predicts the sentence-level dependency arcs, outputs the root label distribution of the file, and obtains the number of root label types of the file to obtain the root label diversity; according to the sentence-level dependency arcs, it obtains the dependency matrix, and solves the maximum dependency depth sentence by sentence to obtain the average dependency depth; according to the root label diversity and the average dependency depth, it obtains the structural complexity coefficient of the file.

[0230] The third module 203 is used to obtain the number of times technical terms appear based on the total number of text characters in the file, and to obtain the technical term density coefficient of the file by combining it with the number of sentences in the file.

[0231] Module 4, 204, is used to obtain the comprehensive complexity coefficient of each file based on the text length coefficient, structural complexity coefficient, and technical terminology density coefficient; and to group the batch of texts according to the comprehensive complexity coefficient to obtain multiple file groups.

[0232] Module 5, 205, is used to generate a corresponding group report for each file group; to concatenate all group reports to obtain a master report; and to perform error analysis on the master report to obtain the final report for batch text processing.

[0233] For specific limitations regarding a batch text processing device, please refer to the limitations of a batch text processing method described above, which will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0234] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a batch text processing method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0235] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0236] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0237] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0238] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0239] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0240] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0241] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended application documents.

Claims

1. A method of processing a batch of text, characterized by, The method comprises the following steps: Obtain a batch of texts, and obtain a text length coefficient of a file according to a total number of text characters of the file; Map each sentence of the file to a vector to obtain a context vector of the file; predict a sentence-level dependency arc based on the context vector of the file, output a root label distribution of the file, and obtain a root label category number of the file to obtain root label diversity; obtain a dependency matrix according to the sentence-level dependency arc, and solve a maximum dependency depth sentence by sentence to obtain an average dependency depth; and obtain a structural complexity coefficient of the file according to the root label diversity and the average dependency depth; Obtain a number of occurrences of a professional term according to the total number of text characters of the file, and obtain a professional term density coefficient of the file in combination with a number of sentences of the file; Obtain a comprehensive complexity coefficient of each file according to the text length coefficient, the structural complexity coefficient and the professional term density coefficient; and group the batch of texts according to the comprehensive complexity coefficient to obtain a plurality of file groups; Generate a corresponding group report for each file group; Splice all group reports to obtain a total report; Perform error analysis on the total report to obtain a final report of the batch of texts; Predict a sentence-level dependency arc based on the context vector of the file, output a root label distribution of the file, and obtain a root label category number of the file to obtain root label diversity, comprising: Based on the context vector of the file, a double-affine attention decoder is used to predict the sentence-level dependency arc, and a normalized exponential function is used to output the root label distribution of the file; According to the root label distribution of the file, a 64-bit rolling hash is performed on the root label of each sentence, and an atomic operation is performed on a GPU end to remove duplicates to obtain the root label category number of the file; According to the root label category number of the file, the root label diversity is obtained; According to the sentence-level dependency arc, a dependency matrix is obtained, and a maximum dependency depth is solved sentence by sentence to obtain an average dependency depth, comprising: According to the sentence-level dependency arc, a dependency matrix is obtained; The dependency matrix is compressed in CSR format and input into a CUDA kernel function to perform parallel BFS, and the maximum dependency depth is solved sentence by sentence; According to the maximum dependency depth, the average dependency depth is obtained.

2. The method of claim 1, wherein, Map each sentence of the file to a vector to obtain a context vector of the file, comprising: Use a regular expression to cut the file according to a sentence-ending punctuation to obtain a plurality of sentences; Map each word in each sentence to a vector through a 128-dimensional FastText to obtain a word vector; Concatenate all word vectors of each sentence into a sequence according to a word order, input a double-layer bidirectional long short-term memory network, and then input a 4-head multi-head attention mechanism to obtain a context representation of each sentence; Obtain the context vector of the file according to all context representations.

3. The method of claim 1 or 2, wherein, Group the batch of texts according to the comprehensive complexity coefficient to obtain a plurality of file groups, comprising: Group the batch of texts according to the comprehensive complexity coefficient to obtain the number of files in different groups; According to the number of files in different groups, set the expected proportion weight vector of each group; according to the comprehensive complexity coefficient and the expected proportion weight vector of each group, group the batch text to obtain a plurality of initial groups; calculate the in-group processing difficulty of each initial group and the in-group splicing error rate of each initial group to obtain the objective function of the current grouping; The random gradient descent algorithm or the grid search algorithm is used to update the comprehensive complexity coefficient and the expected proportion weight vector of each group, and the batch text is re-grouped and the objective function of the current grouping is re-calculated to perform cyclic iteration until a preset condition is met, and the updating and iteration are stopped to obtain a plurality of parameter groups, each of which includes the comprehensive complexity coefficient, the expected proportion weight vector of each group, and the objective function; According to the parameter group where the maximum objective function is located, the optimal comprehensive complexity coefficient and the expected proportion weight vector of each group are obtained to group the batch text to obtain a plurality of file groups.

4. The method of claim 1 or 2, wherein, For each file group, a corresponding group report is generated, including: For each file group, sort the files according to their timestamps to obtain an initial sequence; Calculate the similarity between any two files, and arrange the files with a similarity satisfying a preset condition adjacently to optimize the initial sequence and obtain an optimized sequence; Splice the texts of all files according to the optimized sequence, and insert placeholder markers at the splicing positions to obtain a spliced text; Extract an abstract from the spliced text to obtain the abstract of the spliced text; Perform entity recognition on the spliced text to generate an entity list of the spliced text; Perform keyword recognition on the spliced file to obtain a keyword set of the spliced text; Detect errors in the spliced file and review to obtain an error set of the spliced text; Generate a report according to the index of the abstract, the entity list, the keyword set, and the error set of the spliced text to obtain a group report corresponding to each file group.

5. The method of claim 4, wherein, The similarity between any two files is calculated, including: Obtain a plurality of sample files to obtain a semantic sentence vector of each file; randomly sample and train PCA to obtain a projection matrix and eigenvalues; For any semantic sentence vector, calculate a reduced dimension vector according to the projection matrix; and recalibrate the reduced dimension vector dimension by dimension according to the eigenvalues to obtain a recalibrated vector; Design a weight, and calculate a weighted cosine similarity according to the weight and the recalibrated vector; and linearly map the weighted cosine similarity to obtain the similarity between the two files.

6. An apparatus for processing a batch of text, the apparatus comprising: The method comprises the following steps: A first module is configured to obtain a batch of texts, and calculate a text length coefficient of each file according to the total number of text characters of the file; A second module is configured to map each sentence of the file to a vector to obtain a context vector of the file; predict a sentence-level dependency arc based on the context vector of the file, output a root label distribution of the file, and obtain a number of root label categories to obtain root label diversity; obtain a dependency matrix according to the sentence-level dependency arc, and solve a maximum dependency depth sentence by sentence to obtain an average dependency depth; and obtain a structural complexity coefficient of the file according to the root label diversity and the average dependency depth. The third module is configured to obtain the number of occurrences of the professional terms according to the total number of text characters of the file, and obtain the professional term density coefficient of the file in combination with the number of sentences of the file; The fourth module is configured to obtain the comprehensive complexity coefficient of each file according to the text length coefficient, the structural complexity coefficient and the professional term density coefficient; group the batch of texts according to the comprehensive complexity coefficient to obtain a plurality of file groups; The fifth module is configured to generate a corresponding group report for each file group, splice all the group reports to obtain a total report, and perform error analysis on the total report to obtain a final report of the batch of text processing. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images