A text data automatic collection and intelligent checking system and method

Abstract

The application provides a kind of text data automatic acquisition and intelligent checking system and method, it is related to intelligent document processing technical field, system includes: text feature generation module is identified and parsed text from to be processed data source, generates structured and semantic features;Target text dataset acquisition module is automatically collected according to features to determine target text dataset;Text checking result label group generation module constructs dynamic checking rule library to complete multi-dimensional intelligent checking, generates text checking result label group;Text data checking report generation module carries out quality assessment to target text dataset according to text checking result label group, generates text data checking report.The application solves the technical problems of low efficiency of traditional text data acquisition, fixed checking rule and incomplete data quality evaluation, achieves the automation and intelligentization of text data processing whole process, improves the technical effects of text data acquisition accuracy and checking effectiveness.

Description

Technical Field

[0001] This application relates to the field of intelligent document processing technology, specifically to an automated text data acquisition and intelligent verification system and method. Background Technology

[0002] Text data is the core carrier of data processing, and the quality of its collection and verification directly affects the efficiency and validity of data processing results. Therefore, accurate collection and intelligent verification of text data are crucial in the field of data processing. Current technologies primarily employ traditional data processing methods such as manual extraction and fixed-rule verification to address the problems of text data collection and verification. While these methods have played a significant role in processing single-source structured text data, with the increasing demands of data processing, traditional technologies have revealed numerous limitations when handling multi-source heterogeneous text data. Due to the heterogeneity of text data sources and the complexity of content, traditional data processing methods cannot accurately extract text features and achieve dynamic verification, leading to redundant collected data and distorted verification results, making it difficult to meet the needs for high-quality processing and accurate application of text data. Summary of the Invention

[0003] This application provides an automated text data collection and intelligent verification system and method, which solves the technical problems of low efficiency in traditional text data collection, rigid verification rules, and incomplete data quality assessment. It achieves automation and intelligence in the entire text data processing process, and improves the technical effect of text data collection accuracy and verification effectiveness.

[0004] In view of the above problems, this application provides an automated text data acquisition and intelligent verification system, the system comprising: a text feature generation module, which performs text content recognition based on the data source to be processed, obtains an initial text dataset, parses it, and generates structured features and semantic features; a target text dataset acquisition module, which performs automated acquisition based on the structured features and the semantic features to determine the target text dataset; a text verification result label group generation module, which constructs a dynamic verification rule base to perform multi-dimensional intelligent verification of the target text dataset and generates a text verification result label group; and a text data verification report generation module, which performs quality assessment of the target text dataset based on the text verification result label group and generates a text data verification report.

[0005] On the other hand, this application also provides an automated collection and intelligent verification method for text data. The method includes: performing text content recognition based on the data source to be processed, obtaining an initial text dataset for parsing, and generating structured features and semantic features; performing automated collection based on the structured features and the semantic features to determine the target text dataset; constructing a dynamic verification rule base to perform multi-dimensional intelligent verification on the target text dataset and generating a text verification result label group; and performing quality assessment on the target text dataset based on the text verification result label group and generating a text data verification report.

[0006] In summary, this application obtains dual features by performing content recognition and parsing on the text data source to be processed. After processing such as feature space mapping, vector concatenation, and importance assessment, key text units are selected. A dynamic verification rule base is constructed to carry out multi-dimensional intelligent verification of the target text dataset. Combined with the verification result label group, quality assessment and root cause tracing are performed. This approach accurately processes multi-source heterogeneous text data, significantly improving the accuracy of text data collection and the effectiveness of verification, resulting in higher quality and more reliable text data processing results.

[0007] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a schematic diagram of the structure of the automated text data collection and intelligent verification system of this application.

[0010] Figure 2 This is a flowchart illustrating the automated collection and intelligent verification method for text data in this application.

[0011] Figure labeling: Text feature generation module 10, target text dataset acquisition module 20, text verification result label group generation module 30, text data verification report generation module 40. Detailed Implementation

[0012] This application provides an automated text data acquisition and intelligent verification system and method, which solves the technical problems of low efficiency in traditional text data acquisition, rigid verification rules, and incomplete data quality assessment. It achieves automation and intelligence in the entire text data processing process, and improves the technical effect of text data acquisition accuracy and verification effectiveness.

[0013] Example 1 like Figure 1 As shown in the figure, this application provides an automated text data collection and intelligent verification system, the system comprising: The text feature generation module 10 performs text content recognition based on the data source to be processed, obtains the initial text dataset, parses it, and generates structured features and semantic features.

[0014] In this application, the data source to be processed is various carriers that store text information, including structured carriers such as databases and tables, semi-structured carriers such as XML and JSON, and unstructured carriers such as documents, images, and scanned documents.

[0015] Specifically, the data source to be processed is first determined by type, distinguishing between structured, semi-structured, and unstructured data sources, and text extraction is performed on each type. After obtaining the corresponding original text content, noise filtering is performed to construct an initial text dataset. Then, the layout structure of the dataset is analyzed, and layout and hierarchical features are integrated to form structured features. At the same time, deep text semantic representation data of the dataset is obtained through semantic parsing and used as semantic features. This completes the text content recognition of the data source to be processed, the construction of the initial dataset, and the generation of structured and semantic features. This step will be explained in detail in the following content.

[0016] The target text dataset acquisition module 20 is used to automatically collect data based on the structured features and semantic features to determine the target text dataset.

[0017] Specifically, firstly, a feature space is constructed, and structural features and semantic features are mapped to this feature space to generate corresponding structural feature vectors and semantic feature vectors. Then, the two types of vectors are concatenated to obtain text feature parameters. Subsequently, importance assessment is performed based on these parameters to generate multiple text importance scores. Next, the initial text dataset is traversed according to these scores to complete the selection of key text units. Finally, the selected key text units are aggregated to generate the target text dataset. This step will be explained in detail in the following sections.

[0018] The text verification result label group generation module 30 is used to construct a dynamic verification rule base to perform multi-dimensional intelligent verification on the target text dataset and generate text verification result label groups.

[0019] Specifically, firstly, historical verification logs are used to perform pattern recognition to extract the first type of candidate rules. At the same time, a domain knowledge base is introduced to perform knowledge extraction to obtain the second type of candidate rules. The two types of candidate rules are merged and deduplicated to generate an initial verification rule set. Then, the initial verification rule set is traversed to complete the validity verification. Based on the rule validity parameters, multiple valid rules are selected. Finally, the valid rules are integrated and stored to complete the construction of the dynamic verification rule base. This step will be explained in detail in the following content.

[0020] Next, text recognition and scene recognition are performed on the target text dataset to determine the target text type and business scenario information. Then, the dynamic verification rule base is traversed to dynamically match a subset of verification rules. Subsequently, the target text dataset is sequentially subjected to format, logic, and consistency checks based on the subset of verification rules, and corresponding dimension verification results are generated. After normalizing the verification results of each dimension to obtain the verification record parameters, a text-verification result mapping relationship is constructed based on the parameters. Finally, the verification results of each dimension are encapsulated according to this relationship to generate a text verification result label group. This step will also be explained in detail in the following content.

[0021] The text data verification report generation module 40 performs a quality assessment on the target text dataset based on the text verification result label group and generates a text data verification report.

[0022] Specifically, firstly, multiple verification dimensions are set. Based on the text verification result label group, data transmission verification is completed according to these dimensions, and corresponding verification records are generated. Then, based on the verification records, the data pass rate and data defect rate are calculated. Based on these two indicators, text quality analysis is carried out, and a text quality assessment indicator set is set. Subsequently, the target text dataset is quality assessed according to the indicator set, and a quality compliance rate is generated. Finally, root cause tracing is performed based on the quality compliance rate, and the identified defect analysis data is added to the text data verification report, thereby completing the quality assessment and verification report generation of the target text dataset. This step will be explained in detail in the following content.

[0023] In one possible implementation, the text feature generation module 10 further includes: The system includes the following components: a data source type information acquisition unit, used to determine the data source type of the data source to be processed, including structured data sources, semi-structured data sources, and unstructured data sources; a first original text content acquisition unit, used to perform text extraction operations based on the structured data source to obtain first original text content; a second original text content acquisition unit, used to perform text extraction operations based on the semi-structured data source to obtain second original text content; a third original text content acquisition unit, used to perform text extraction operations based on the unstructured data source to obtain third original text content; an initial text dataset construction unit, used to perform noise filtering on the first, second, and third original text contents to construct an initial text dataset; a text feature determination unit, used to perform layout structure analysis on the initial text dataset to determine text layout features and hierarchical features; a structured feature construction unit, used to integrate the text layout features and hierarchical features to construct structured features; and a semantic feature acquisition unit, used to perform semantic parsing on the initial text dataset to determine deep text semantic representation data, and use the deep text semantic representation data as semantic features.

[0024] Specifically, the first step is to determine the data source type. This involves identifying the storage format and data organization of the data source to be processed using data format recognition. This distinguishes between structured data sources, semi-structured data sources, and unstructured data sources. Structured data sources are those with a fixed two-dimensional table structure, semi-structured data sources are those with some markers that define the meaning of the data, and unstructured data sources are plain text or image data sources without a fixed data organization. After the determination is completed, the corresponding data source type information is identified.

[0025] For different types of data sources after identification, targeted text extraction operations are performed. For structured data sources, the text field content in the data table is directly read, and non-text information such as field identifiers is removed to obtain the first original text content. For semi-structured data sources, the markers in the data source are identified. First, the data source content is traversed using a character matching algorithm to identify various markers that define the meaning of the data and mark them for location. Then, according to the hierarchy and pairing rules of the markers, the text content between each marker is extracted, and the markers themselves and the invalid identifier information attached to the markers are removed to extract the valid text content, thus obtaining the second original text content. For unstructured data sources, image preprocessing is performed on image-type unstructured data sources. After image noise reduction, tilt correction, and character region localization, the characters in the image are converted into editable text format using an optical character recognition algorithm. Then, a unified text traversal is performed on the editable text converted from the image and plain text-type unstructured data sources, and the valid text content is extracted segment by segment. The text content of all unstructured data sources is extracted in batches to obtain the third original text content.

[0026] Next, after obtaining the three types of original text content, noise filtering is used to uniformly process the first, second, and third types of original text content. First, a character feature recognition algorithm is used to traverse all characters in the three types of original text content. According to character encoding rules, garbled characters are identified. According to character whitespace attributes, consecutive whitespace characters and meaningless whitespace placeholders are identified. According to character repetition rules, consecutive repeated invalid characters and text fragments are identified. Then, according to the effective expression rules of text content, all noise information such as identified garbled characters, whitespace characters, and repeated invalid characters is deleted. At the same time, the effective punctuation and normal format intervals of the text itself are preserved. After noise filtering is completed, the three types of text content are classified and labeled according to the data source type and then integrated and summarized to construct the initial text dataset, ensuring the effectiveness and cleanliness of the text content in the initial text dataset.

[0027] Next, a layout structure analysis was performed on the constructed initial text dataset. The text in the initial text dataset was analyzed segment by segment to determine text layout features such as line spacing, paragraph division, and the positional relationship between headings and body text. At the same time, the hierarchical features of the text were determined based on the heading level and content hierarchy. The identified text layout features and hierarchical features were then classified and integrated. According to the preset feature integration format of key-value pairs, the types of layout features and hierarchical features were associated with the corresponding content and stored to construct structured features, thereby realizing the quantitative extraction of the external form features of the text.

[0028] While constructing structured features, semantic parsing is performed on the initial text dataset. First, all text content in the initial text dataset is split into sentences. Then, word segmentation and grammatical structure recognition are carried out on the split sentences in sequence: each character in the sentence is traversed, and character matching is performed according to the preset general vocabulary library. This general vocabulary library adopts the commonly used general Chinese word segmentation vocabulary library in this field, including Harbin Institute of Technology LTP Chinese vocabulary library, Baidu LAC Chinese vocabulary library, etc. Continuously matching characters are combined into words, and modal particles and conjunctions without actual semantic meaning are removed to extract the core words in the sentence. At the same time, according to the grammatical rules of subject-verb-object and attributive-adverbial-complement, the collocation relationship and order between the core words are sorted out to clarify the grammatical structure of the text.

[0029] After completing word segmentation and grammatical structure recognition of all texts, the extracted core words are matched word-by-word with a pre-defined general text semantic database. This semantic database uses publicly available general Chinese semantic knowledge bases, including the CNKI HowNet semantic knowledge base and the extended version of the Chinese thesaurus, which contain semantic correspondences and basic semantic logical relationships of general words. During matching, the basic semantics of words in the semantic database are retrieved first, and then the semantic relationships between words are sorted out in combination with the grammatical structure of the text to explore the actual meaning and internal logical relationships behind the text. Subsequently, each core word is converted into a fixed-dimensional word semantic vector. The conventional value range of this fixed dimension is 50 to 300 dimensions. In general scenarios, a fixed dimension of 100 or 200 dimensions is used. The dimension value can be linearly adjusted within this range according to the accuracy requirements of text processing. At the same time, the semantic relationships between words and the semantic connection relationships between sentences are sorted out and recorded to form text semantic relationship information. The above word semantic vectors and text semantic relationship information are integrated into deep text semantic representation data. This deep text semantic representation data is directly used as semantic features to achieve effective extraction of the inherent semantic features of the text.

[0030] By performing targeted text extraction and noise filtering on different types of data sources, and combining layout structure analysis and semantic parsing methods to extract the structural and semantic features of the text respectively, a unified identification and parsing of multi-source heterogeneous data sources to be processed is achieved, effectively extracting the dual features of the text.

[0031] In one possible implementation, the target text dataset acquisition module 20 further includes: The system includes a feature vector generation unit, which constructs a feature space, maps the structured features to the feature space to generate a structured feature vector, and maps the semantic features to the feature space to generate a semantic feature vector. A text feature parameter acquisition unit is used to concatenate the structured feature vector and the semantic feature vector to obtain text feature parameters. A text importance score generation unit performs importance evaluation based on the text feature parameters to generate multiple text importance scores. A key text unit acquisition unit is used to traverse the initial text dataset according to the multiple text importance scores to filter and determine key text units. A target text dataset acquisition unit is used to aggregate the key text units to generate the target text dataset.

[0032] Specifically, the first step is to construct the feature space. First, summarize the feature types of all structured and semantic features generated in the previous steps. Then, sort out the number of sub-items of text layout features and hierarchical features included in the structured features, as well as the number of dimensions of lexical semantic vectors and text semantic association information included in the semantic features. Next, count the value range of each sub-feature, set a unified feature quantization benchmark, and convert all non-numerical feature content into computable numerical data. Then, use the sum of the total quantization dimensions of the structured features and the total quantization dimensions of the semantic features as the total dimension of the feature space, and construct a multi-dimensional Euclidean space as the feature space. At the same time, divide the feature space into fixed dimension intervals. Set the preorder interval as the mapping interval for structured features and the postorder interval as the mapping interval for semantic features. Clarify the feature sub-items and mapping rules corresponding to each dimension interval, and complete the construction of the feature space.

[0033] Next, feature mapping is performed. For structured features, text layout features and hierarchical features are first converted into numerical data of the corresponding dimensions according to a preset quantization benchmark. Specifically, line spacing, paragraph division, and the positional relationship between headings and body text in text layout features are converted into corresponding spacing values, paragraph number values, and positional percentage values, respectively. Heading levels and content subordination in hierarchical features are converted into corresponding hierarchical number values ​​and subordination weight values, respectively. Then, all the converted values ​​are sequentially filled into the corresponding dimensional positions according to the dimensional order of the structured feature-specific mapping interval in the feature space to generate a structured feature vector that matches the dimension of the feature space. For semantic features, the word semantic vectors are first dimensionally aligned to ensure that their dimensions are consistent with the corresponding sub-dimensions of the semantic feature-specific mapping interval. Then, the text semantic association information is converted into corresponding semantic association weight values ​​and filled into the corresponding dimensional positions of the semantic feature-specific mapping interval to generate a semantic feature vector that matches the dimension of the feature space.

[0034] Then, a vector concatenation operation is performed. According to the preset dimensional order of the feature space, all dimensional values ​​of the structural feature vector are arranged in complete order. Then, all dimensional values ​​of the semantic feature vector are arranged at the end of the structural feature vector to form a one-dimensional feature vector that is completely consistent with the total dimension of the feature space. This one-dimensional feature vector is used as the text feature parameter of the corresponding text segment. For each continuous text segment in the initial text dataset, a unique corresponding text feature parameter is generated, thereby completing the feature parameter extraction of all text segments.

[0035] Next, the initial text dataset is divided into multiple text units, each containing a continuous text segment. Each text unit is assigned a corresponding unit identifier. Then, text feature parameters are matched according to the unit identifier, and multiple text feature vectors are extracted. Finally, deep learning is performed on each text unit based on the text feature vectors to complete the importance evaluation and generate multiple corresponding text importance scores. This step will be explained in detail later.

[0036] After generating all text importance scores, a text filtering operation is performed. First, a qualified threshold for text importance is preset. The text importance scores corresponding to all text segments in the initial text dataset are summarized to form a complete score set. Then, all scores within the score set are sorted in ascending order to obtain an ordered score sequence. Simultaneously, the total number of text segments corresponding to this ordered score sequence is counted, completing the basic data preparation. Next, based on the actual business needs of text collection, a target retention ratio for core text is preset. In general scenarios, an 80% core text retention ratio is used by default. Those skilled in the art can flexibly adjust it within the range of 50% to 90% according to actual collection needs. Simultaneously, the corresponding filtering quantile value is calculated based on the target retention ratio. Then, in the ordered score sequence, the score position corresponding to this filtering quantile value is located, and the text importance score at that position is determined as the preliminary qualified threshold. After calculating the preliminary qualified threshold, the number of text segments that can be retained corresponding to this threshold is counted. If the number of retained segments exceeds or falls below the business expectation range, the target retention ratio is directly adjusted and recalculated to complete the final preset text importance qualified threshold.

[0037] Then, following the original order of the text segments in the initial text dataset, the text importance score corresponding to each text segment is traversed sequentially. Text segments with a text importance score greater than or equal to a preset text importance threshold are marked as candidate key text units, while text segments with a text importance score lower than the preset threshold are marked as invalid content and removed. After the traversal is completed, all the remaining candidate key text units are determined as key text units. At the same time, the original position information, corresponding text feature parameters, and text importance score of each key text unit in the initial text dataset are recorded synchronously.

[0038] Finally, an aggregation process is performed. Based on the original position information of each key text unit, all key text units are sorted in ascending order according to the original chronological order of the initial text dataset to restore the normal reading logic of the text content. Then, the continuous text segments of each key text unit are joined together in the sorted order to form the complete target text content. At the same time, the original position information, text feature parameters, and text importance scores corresponding to each key text unit are associated and bound with the joined target text content. Finally, the associated and bound complete text content and corresponding data are integrated and stored to generate the target text dataset.

[0039] By completing the entire process of feature space construction, feature mapping and vector concatenation, text importance metric evaluation, key text filtering and ordered aggregation in steps, the system achieves the effect of accurately filtering core text content based on the structured and semantic features of text, and efficiently completing the automated collection of target text datasets.

[0040] In one possible implementation, the text importance score generation unit further includes: The system includes several sub-units: a sub-unit for obtaining multiple text units, which divides the initial text dataset into multiple text units, each containing a continuous text segment; a sub-unit for obtaining multiple unit identifiers, which assigns identifiers to the multiple text units based on the continuous text segments to determine multiple unit identifiers; a sub-unit for obtaining text feature vectors, which matches the text feature parameters according to the multiple unit identifiers to extract multiple text feature vectors; and a sub-unit for obtaining text importance scores, which performs deep learning on the multiple text units based on the multiple text feature vectors, evaluates the importance of the multiple text units based on the learning results, and obtains multiple text importance scores.

[0041] Specifically, the process begins by acquiring the complete text content of the initial text dataset, identifying paragraph separators, line breaks, and sentence-ending punctuation marks within the text, and initially splitting the complete text content based on natural paragraphs. Then, the continuity of text within each initially split paragraph is checked to ensure there are no interruptions in the text within each paragraph. Simultaneously, a preset range of characters for a single text unit is established, typically between 50 and 500 characters. If the number of characters in a initially split paragraph exceeds the preset upper limit, the paragraph is split a second time according to sentence-ending punctuation. If the number of characters in a initially split paragraph is below the preset lower limit, adjacent consecutive short paragraphs are merged. Finally, the initial text dataset is split into multiple text units, each containing one and only one consecutive text segment. The original start and end positions of each text unit in the initial text dataset are recorded simultaneously.

[0042] Next, based on the continuous text segments corresponding to each text unit obtained from the splitting, a unique unit identifier is assigned to each text unit. The unit identifier is generated by combining a continuous numerical sequence number with a fixed-length positional code. First, according to the original order of each text unit in the initial text dataset, a sequentially increasing continuous Arabic numeral sequence number is assigned to each text unit. Then, the values ​​of the original start and end positions corresponding to each text unit are converted into fixed-length positional codes. The numerical sequence number and the positional code are concatenated to form a unique unit identifier for each text unit. At the same time, each unit identifier is associated and bound with the continuous text segments and original positional information of the corresponding text unit to generate an identifier-text mapping table. This completes the identifier assignment for all text units and determines multiple unit identifiers that correspond one-to-one with each text unit.

[0043] After all text units are assigned identifiers, text feature parameter matching and text feature vector extraction are performed according to the generated multiple unit identifiers. First, the text feature parameters generated in the previous steps and corresponding to the initial text dataset are obtained. Each text feature parameter is associated with the original position information of the corresponding text segment. Then, all generated unit identifiers are traversed, and the original position information of the text unit corresponding to each unit identifier is extracted based on the identifier-text mapping table. The original position information is matched one by one with the position information associated with the text feature parameters. When the overlap of position information reaches 100%, the text feature parameter is determined to be the exclusive feature parameter of the text unit corresponding to the current unit identifier. After all unit identifiers and text feature parameters are matched, the one-dimensional feature vector formed by concatenating the structural feature vector and the semantic feature vector is extracted from each matched text feature parameter. This one-dimensional feature vector is used as the text feature vector of the corresponding text unit. Finally, multiple text feature vectors corresponding one-to-one with multiple text units are obtained, and each text feature vector is associated with and stored with the unit identifier of the corresponding text unit.

[0044] Next, we performed the selection and basic adaptation of the pre-trained model. We chose the publicly available, lightweight, and easily deployable TextCNN pre-trained model, which is a general pre-trained model for text feature learning and regression tasks. The complete pre-trained weights and basic network architecture can be obtained directly through public deep learning frameworks such as TensorFlow and PyTorch. We first loaded the complete network structure and pre-trained weights of the pre-trained model, and retained the network structure and pre-trained weight parameters of the core feature extraction layers such as convolutional layers and pooling layers inside the model without modification. We only adapted and adjusted the input and output layers of the model. We set the dimension of the model input layer to be completely consistent with the dimension of the extracted text feature vector, so that the input layer can directly receive the text feature vector of the corresponding dimension. At the same time, we adjusted the original multi-class output layer of the model to a single-neuron regression output layer, and used the Sigmoid function as the activation function so that the model output can output continuous values ​​in the range of 0 to 1, thus completing the basic adaptation of the model.

[0045] After completing the basic adaptation of the pre-trained model, fine-tuning training of the model is carried out. First, a publicly available text importance annotation dataset in this field is obtained. This dataset contains manually annotated text segments, corresponding text feature vectors, and importance annotation scores. The annotation scores are continuous values ​​in the range of 0 to 1, with higher values ​​indicating higher importance of the corresponding text. The annotation dataset is divided into training and validation sets in an 8:2 ratio. The text feature vectors in the training set are input into the adapted pre-trained model. The annotation importance scores are used as the baseline ground truth. Mean squared error is used as the loss function, and the Adam optimizer is used for iterative training. The number of training iterations is preset to 10. After each iteration, the prediction accuracy of the model is verified using the validation set. When the loss value of the validation set tends to stabilize and no longer decreases, the training is terminated early, and the weight parameters of the trained model are saved to obtain the adapted text importance evaluation model.

[0046] Finally, importance assessment and score generation are performed. Multiple text feature vectors, each corresponding to a text unit, are sequentially input into the adapted text importance assessment model according to the order of their unit identifiers. The model uses its internally retained core feature extraction layer to perform deep feature learning on the input text feature vectors, mining structural and semantic features related to text importance. Then, the adjusted output layer outputs a continuous value between 0 and 1 for each text feature vector. This output value is multiplied by 100 to convert it into a text importance score between 0 and 100. Finally, multiple text importance scores are obtained, each corresponding to a text unit. At the same time, each text importance score is associated and bound to the unit identifier and text feature vector of the corresponding text unit for storage.

[0047] By completing the entire process of text unit standardization, unique unit identifier allocation, accurate text feature vector matching and extraction, pre-trained model adaptation and fine-tuning, and importance score inference output step by step, the system achieves the effect of accurately and efficiently generating the text importance score corresponding to each text unit based on text feature vectors through deep learning.

[0048] In one possible implementation, the text verification result tag group generation module 30 further includes: The first candidate rule acquisition unit is used to schedule historical verification logs for pattern recognition and extract the first type of candidate rules; the second candidate rule acquisition unit is used to introduce a domain knowledge base for knowledge extraction and extract the second type of candidate rules; the initial verification rule set acquisition unit is used to merge and deduplicate the first type of candidate rules and the second type of candidate rules to generate an initial verification rule set; the multiple valid rule determination unit is used to traverse the initial verification rule set for validity verification, filter the initial verification rule set according to the rule validity parameters, and determine multiple valid rules; the dynamic verification rule base construction unit is used to integrate and store the multiple valid rules to construct the dynamic verification rule base.

[0049] Specifically, historical verification logs containing complete verification processes and manual review conclusions are retrieved from the system. Redundant log entries without valid verification results or review confirmations are first removed, retaining only valid logs that have been verified in practice. Using keyword matching and fixed defect template matching, pattern recognition is performed on the valid logs. Reusable defect judgment logic and compliance verification standards are extracted from each log entry, and each logic is organized into standardized rule entries containing the verification object, judgment conditions, and compliance requirements. After extraction, the first category of candidate rules is obtained.

[0050] Simultaneously, authoritative domain knowledge bases for the corresponding business areas are introduced, including publicly available authoritative content such as industry standards, compliance specifications, and statutory verification criteria. The knowledge base content is first broken down by chapters and clauses, and the mandatory compliance requirements, format and content constraints are extracted using keyword matching. The compliance requirements expressed in natural language are converted into standardized rule entries containing verification content, judgment criteria, and anomaly identification rules. After extraction, the second type of candidate rules is obtained.

[0051] Next, the first and second categories of candidate rules are summarized. Rules are grouped according to three core dimensions: format validation, logical validation, and consistency validation. Within each group, the validation objects and judgment conditions of each rule are compared, eliminating rule entries with completely duplicate validation logic and retaining only the one with the most complete information. Rules with overlapping validation scopes are merged into a single rule entry with complete coverage. A unique rule number is assigned to all retained rules, forming the initial validation rule set.

[0052] The three pre-defined validation criteria for rule validity are: having clearly defined and executable input / output logic, a detection rate of at least 85% for known defects, and a false positive rate of at least 5% for compliant text. A set of validation texts, manually reviewed and annotated, is prepared. This set includes both known defect texts and standard compliant texts. In general scenarios, the minimum sample size for the validation text set is at least 200 texts, with a ratio of at least 1:1 between known defect texts and standard compliant texts. The sample size can be increased synchronously according to the coverage of the validation business. The initial validation rule set is traversed sequentially by rule number. The validation text sets are input into the corresponding rules to complete the validation. The three validation indicators for each rule are calculated, generating rule validity parameters. All rules that meet the validation criteria are retained as valid rules.

[0053] Finally, all valid rules are classified and archived in three levels according to applicable text type, business scenario, and verification dimension. A rule file containing rule number, source, scope of application, verification logic, and verification parameters is created for each valid rule. The rule file is then associated and bound with the corresponding verification logic and stored in a database that supports dynamic read and write. At the same time, a search index covering applicable scenarios, text types, and verification dimensions is created. Dynamic adjustment interfaces for four basic operations—adding rules, updating versions, temporarily disabling rules, and permanently removing rules—are reserved. Each type of operation is equipped with input parameter format verification and rule validity pre-verification logic to ensure data consistency and operational stability after the rule library is updated, thereby completing the construction of a dynamic verification rule library.

[0054] By extracting candidate verification rules from both historical verification logs and authoritative domain knowledge bases, and then filtering out stable and usable valid rules through merging, deduplication, standardization, and multi-dimensional validity verification, the entire process of systematic classification, archiving, and standardized storage is completed. This achieves the goal of building a dynamic verification rule library that is tailored to actual business needs, meets industry compliance requirements, and is dynamically adjustable and stably executed.

[0055] In one possible implementation, the text verification result tag group generation module 30 further includes: The system comprises the following components: a target text type acquisition unit, which performs text recognition based on the target text dataset to determine the target text type; a business scenario information acquisition unit, which performs scenario recognition based on the target text dataset to determine business scenario information; a verification rule subset acquisition unit, which performs dynamic matching by traversing a dynamic verification rule library according to the target text type and the business scenario information to determine a verification rule subset; a first-dimensional verification result acquisition unit, which performs format verification on the target text dataset according to the verification rule subset to generate a first-dimensional verification result; and a second-dimensional verification result acquisition unit, which performs logical verification on the target text dataset according to the verification rule subset to generate a second-dimensional verification result. The system includes: a verification result acquisition unit for obtaining third-dimensional verification results; a verification record parameter acquisition unit for normalizing the first-dimensional verification results, the second-dimensional verification results, and the third-dimensional verification results to obtain verification record parameters; and a text verification result tag group generation unit for performing text mapping based on the verification record parameters, constructing a text-verification result mapping relationship, and encapsulating the first-dimensional verification results, the second-dimensional verification results, and the third-dimensional verification results according to the text-verification result mapping relationship to generate the text verification result tag group.

[0056] Specifically, after loading the automatically collected target text dataset, the text content is retrieved and matched using a pre-set text type feature library. This feature library is pre-built by aggregating exclusive keywords and standard structural features of common text types in this field, covering four core types: government documents, business contracts, technical specifications, and compliance application materials. Each type of text corresponds to no less than 50 exclusive keywords and 3 core structural features. The overall overlap between the text and each type of feature is statistically analyzed through keyword matching. In general scenarios, the preset overlap threshold is 60%. The type with the highest overlap that meets the threshold requirement is determined as the target text type. If the overlap of all types does not reach the threshold, manual review and confirmation are triggered, and the final matching result is recorded simultaneously.

[0057] Simultaneously, business scenario identification is performed on the target text dataset. A pre-set business scenario feature library is used for matching. This feature library is pre-built by collecting exclusive keywords and text usage features of common business scenarios, covering four core scenarios: official document release and review, contract compliance verification, technical document archiving, and application material pre-review. Each scenario corresponds to no less than 30 exclusive keywords. Combined with the determined target text type, the core keywords and execution entity information related to the business scenario are extracted from the text, and the comprehensive matching degree of the scenario is calculated. The pre-set qualified matching degree threshold is 60% under general scenarios. The scenario with the highest matching degree and meeting the threshold requirement is determined as the suitable business scenario, and the corresponding business scenario information is generated. At the same time, the business scenario information is associated and bound with the target text type.

[0058] Next, the dynamic validation rule base that has been built in the previous steps is retrieved. Relying on the multi-dimensional search index preset in the base, the rule base is first filtered by the target text type as the first filtering dimension, and the rule entries with applicable scope covering the text type are filtered out. Then, the business scenario information is used as the second filtering dimension, and the rule entries corresponding to the business scenario are further matched in the first round of filtering results. Invalid rules marked as temporarily disabled or expired are removed at the same time. Finally, the valid rules obtained by filtering are integrated into a subset of validation rules. At the same time, each rule in the subset is labeled with the corresponding validation dimension, which are three categories: format validation, logic validation, and consistency validation.

[0059] Then, based on the subset of validation rules, format validation, logical validation, and consistency validation are performed on the target text dataset. The specific steps are as follows: Extract the rule entries marked as format validation from the subset of validation rules, and clarify the validation content, judgment criteria, and pass conditions for each rule. Validate the target text dataset content in the order of the rules. The validation scope covers common format requirements such as font format, heading hierarchy, paragraph layout, punctuation, completeness of required fields, and page numbering. A single rule is considered compliant if it fully meets the pass conditions, and deemed defective if it does not. Record the validation results for each rule, including whether a defect was found, the precise location of the defect in the text, and details of the defect content. After all format rule validations are completed, summarize the defect records and bind them to the corresponding text fragments to generate the first dimension validation results.

[0060] After format validation, rules marked as logical validation are extracted from the validation rule subset. The validation logic, anomaly identification criteria, and pass conditions are then analyzed for each rule. Logical validation is performed on the target text dataset. First, the text content is split sentence by sentence. After sentence-by-sentence splitting, a dependency parsing algorithm is used to generate the grammatical dependency tree structure for each sentence, extracting the core elements of subject, predicate, and object, and constructing sentence-level semantic relation pairs. Simultaneously, sentence-level semantic vector representations are generated based on the lexical semantic vectors from the aforementioned semantic features. Cosine similarity is calculated for the sentence-level semantic vectors of adjacent sentences. Here, A and B are the semantic vectors of adjacent sentences, and ||A|| and ||B|| are the Euclidean norms of the corresponding semantic vectors. A semantic contradiction is defined as follows: when the semantic vector similarity between adjacent sentences is less than 0.3, and the semantic direction pointed to by the causal or adversative conjunction is opposite to or unrelated to the semantic vector direction of the preceding and following sentences, it is considered a semantic contradiction. A logical deficiency is defined as the absence of any legally necessary constituent element (subject, verb, object), which is determined by the element list of the corresponding text type in the preset rule base. A logical conflict is defined as the inconsistency in time, quantity, or subject information between sentences, with a numerical difference exceeding 10%. The semantic connections, causal relationships, and constituent elements of the context are analyzed. The text is then checked against the rules for semantic contradictions, inverted causal logic, missing legal elements, and ambiguous content. A rule check that does not find any corresponding anomalies is considered compliant; anomalies are considered defects. The location, type, and details of each logical defect are recorded simultaneously. After all logical rule checks are completed, the defect records are summarized and bound to the corresponding text fragments to generate the second-dimensional check results.

[0061] After completing the logical verification, extract the rule entries marked as consistency verification from the subset of verification rules, clarify the verification scope, comparison standards and passing conditions for each rule, and conduct consistency verification on the target text dataset. The verification scope covers the consistency of professional terminology, key contextual data, title and body content, attachment and body content, and cited content and original text. First, extract professional terminology, key data, title entries, and cited content from the text to establish a full-text comparison benchmark library. Then, scan the text content segment by segment and compare it with the benchmark library one by one. If the content of a single rule verification completely matches the benchmark library, it is judged as compliant. If there are differences, it is judged as defective. Identify and record the location and details of inconsistent abnormal content. After completing all consistency rule verifications, summarize the abnormal records and bind them with the corresponding text segments to generate the third-dimensional verification results.

[0062] After completing the verification across the three dimensions, the raw data of the verification results for each dimension is obtained. First, all data is formatted uniformly, converting defect records from different dimensions into a standardized structure containing five core fields: defect location, defect type, defect level, rule source, and defect details. Then, defects are categorized according to their impact on text compliance and usability. Severe defects violate mandatory standards and cause the loss of core text effectiveness, corresponding to a quantitative score of 10 points. General defects affect text rigor but do not change core effectiveness, corresponding to a quantitative score of 3 points. Minor defects are minor flaws that do not affect content comprehension, corresponding to a quantitative score of 1 point. Simultaneously, the total number of verification entries, the number of compliant entries, the total number of defects detected, the number of defects at each level, and the corresponding total score are statistically analyzed. The standardized defect records are then integrated with the statistical data to obtain the verification record parameters.

[0063] Finally, extract the text location information corresponding to each defect record, accurately associate each defect record with the text fragment at the corresponding location, establish a one-to-one correspondence between each verification result and the corresponding text fragment, forming a complete text-verification result mapping relationship. Based on this mapping relationship, sort and integrate the verification results of the three dimensions according to the original chronological order of the text fragments, generate a verification result label for each text fragment containing compliance status, number of defects, defect level, and defect details, summarize and encapsulate the verification result labels of all text fragments, generate a text verification result label group that corresponds one-to-one with the content of the target text dataset, and store the label group in association with the target text dataset.

[0064] By completing the full-link operation of accurately identifying the target text type and business scenario step by step, dynamically matching the subset of adaptive verification rules, performing closed-loop verification across three dimensions, and standardizing and normalizing the verification results and encapsulating them with tags, the system achieves the effect of performing multi-dimensional accurate and intelligent verification of the target text dataset based on a dynamic verification rule library, generating a standardized text verification result tag group that corresponds one-to-one with the text content.

[0065] In one possible implementation, the text verification result tag group generation unit further includes: The defect type determination subunit is used to continuously monitor the text verification result tag group and determine the defect type based on the monitoring results; the automatic update mechanism execution subunit is used to capture the occurrence frequency parameter of the defect type, and trigger the automatic update mechanism when the occurrence frequency parameter exceeds a preset frequency threshold; the verification result generation subunit is used to construct a first candidate verification rule based on the defect type matching text unit features through the automatic update mechanism, perform effective verification based on the first candidate verification rule, and generate a verification result; the initial verification rule set update subunit is used to fill the first candidate verification rule into the initial verification rule set for update and optimization when the verification result passes.

[0066] Specifically, the generated text verification result tag group is obtained, and all defect records within the tag group are continuously collected according to a preset fixed monitoring cycle. In general scenarios, the monitoring cycle is set to be synchronized with a single batch of text verification tasks. After each batch of text verification tasks is completed, the collected defect record data is updated synchronously. Alternatively, it can be set to collect data at a fixed cycle of natural days or natural weeks according to business needs. For all collected defect records, a preset defect classification system with the same verification dimensions as the previous steps is used to complete the type labeling. This classification system is divided into three core categories: format defects, logic defects, and consistency defects. Each core category corresponds to a sub-type of defect that matches the verification rules one by one. By using keyword matching, each defect record in the tag group is mapped to a unique defect sub-type, completing the type classification of the defect records and finally determining the defect type corresponding to each defect.

[0067] Next, within a preset statistical period, statistics are grouped and analyzed by defect type. The statistical period is consistent with the monitoring period. The total number of occurrences of each defect type within the statistical period is calculated, and this total number is determined as the occurrence frequency parameter for the corresponding defect type. A preset defect frequency threshold is set to 5 times by default in general scenarios. Those skilled in the art can flexibly adjust it within the range of 3 to 10 times according to the strictness of business verification. The occurrence frequency parameter of each defect type is compared with the preset frequency threshold one by one. When the occurrence frequency parameter of a certain defect type is greater than or equal to the preset frequency threshold, the automatic update mechanism is immediately triggered, and the defect type is marked as a high-frequency defect type to be optimized.

[0068] After the automatic update mechanism is triggered, the first step is to retrieve the defect records corresponding to the high-frequency defect types to be optimized. The verification logic for this defect type is then compared with existing rules in the dynamic verification rule library and the initial verification rule set. If the existing rules fully cover the verification logic for this defect type, the automatic update process terminates. If the existing rules do not cover or only partially cover the verification logic for this defect type, the subsequent process continues. Then, the text unit features associated with each defect record are extracted one by one, including the target text type of the defect text, business scenario information, content features of the corresponding text fragment, the specific scenario that triggered the defect, and the core basis for anomaly judgment. The extracted text unit features and the judgment logic of high-frequency defects are matched with the standardized structure of existing rules in the dynamic verification rule library, converting them into standardized rule entries containing the verification object, scope of application, judgment conditions, pass criteria, and anomaly identification rules, generating the first candidate verification rule. Subsequently, using the same validity verification standard as the initial verification rule set construction step mentioned above, a verification dataset containing the high-frequency defect text and the standard compliant text is prepared. In general scenarios, the minimum sample size of the verification dataset is no less than 200, and the ratio of high-frequency defect text to standard compliant text is no less than 1:1. The first candidate verification rule is input into the verification dataset to complete the verification. The defect detection rate and false positive rate of the rule are calculated. Compared with the preset verification standard of a detection rate of no less than 85% and a false positive rate of no more than 5%, a verification result of verification pass or verification fail is generated.

[0069] When the generated verification result meets the preset pass criteria, a unique rule number is assigned to the first candidate verification rule. Simultaneously, basic information such as the rule's source, applicable scenarios, verification parameters, and version number is updated. Then, this candidate rule is compared line by line with existing rules in the initial verification rule set. Existing invalid rules that completely duplicate the candidate rule's verification logic or overlap with its verification scope are eliminated to avoid rule redundancy. After deduplication, the first candidate verification rule is officially written into the initial verification rule set, completing the content update and version upgrade of the initial verification rule set. Simultaneously, the updated initial verification rule set is synchronized to the dynamic verification rule library, completing the rule library's classification and archiving, retrieval index updates, and version synchronization, ensuring that subsequent text verification stages can directly call the latest updated rules.

[0070] By continuously monitoring the label groups of text verification results and accurately identifying high-frequency defect types, an automatic update mechanism is triggered after pre-deduplication verification. This completes the standardized construction of candidate verification rules and the validity verification of unified standards. Ultimately, it achieves synchronous iterative optimization of the initial verification rule set and the dynamic verification rule library, thereby continuously improving the defect coverage of verification rules, reducing the missed detection rate of similar text defects, and ensuring that the text verification system continuously adapts to business needs.

[0071] In one possible implementation, the text data verification report generation module 40 further includes: The data transmission verification record acquisition unit is used to set multiple verification dimensions, perform data transmission verification based on the text verification result label group according to the multiple verification dimensions, and generate data transmission verification records; the data parameter acquisition unit is used to perform data calculations according to the data transmission verification records to obtain the data pass rate and data defect rate; the text quality assessment index set construction unit performs text quality analysis based on the data pass rate and the data defect rate to set a text quality assessment index set; the quality compliance rate acquisition unit is used to perform quality assessment on the target text dataset according to the text quality assessment index set and generate a quality compliance rate; the defect analysis data acquisition unit performs root cause tracing based on the quality compliance rate and determines that defect analysis data should be added to the text data verification report.

[0072] Specifically, firstly, three core verification dimensions are defined, corresponding one-to-one with the multi-dimensional intelligent verification steps mentioned above: format verification dimension, logic verification dimension, and consistency verification dimension. Each dimension forms a one-to-one mapping relationship with the corresponding dimension verification results and subset of verification rules within the text verification result label group, ensuring that the dimension division is completely consistent with the preceding verification steps. Then, the generated text verification result label group is retrieved, and the verification data within the label group is split and aggregated according to the three preset dimensions. Data transmission verification is performed item by item. Core verification items include the completeness of the unit identifier corresponding to each verification result, the consistency between defect information and the corresponding verification rule, the accuracy of text location information, and the traceability of verification results. Invalid verification data without corresponding unit identifiers or matching verification rules are removed. For each dimension verification completed, the total number of valid verification items, the number of compliant items, the number of defect-detected items, and the amount of invalid data removed for that dimension are recorded simultaneously. Finally, the verification details for all dimensions are summarized to generate a complete data transmission verification record.

[0073] Next, the generated data transmission verification records are retrieved, and the total number of valid verification entries, the number of compliant entries, and the number of defect-detected entries are extracted across three dimensions. The statistical standards are then standardized to match those used in the verification process described earlier. The total number of valid verification entries is calculated as the product of the total number of valid rules within the subset of verification rules and the total number of text units in the target text dataset, ensuring consistent and unbiased statistical standards throughout the entire process. Subsequently, quantitative calculations are performed using fixed formulas. The data pass rate is calculated by multiplying the number of compliant entries by the total number of valid verification entries by 100%, with individual pass rates for each verification dimension and overall pass rates for all dimensions calculated separately. The data defect rate is calculated by multiplying the number of defect-detected entries by the total number of valid verification entries by 100%, with individual defect rates for each verification dimension and overall defect rates for all dimensions calculated separately. After completing the calculations for all data, the calculation process and detailed results are retained to ensure that all data is verifiable and traceable.

[0074] Based on the comprehensive pass rate and defect rate of the completed data across all dimensions, as well as the pass rate and defect rate data of each individual dimension, a text quality weakness analysis is conducted to pinpoint the quality performance of the target text dataset across different verification dimensions, identifying the core weak dimensions with high defect rates and low pass rates. Combining the text quality analysis results, a text quality assessment indicator set is constructed, corresponding one-to-one with each verification dimension. This indicator set has a two-tier structure of primary and secondary indicators. The primary indicators are three core indicators: format compliance, logical rigor, and content consistency, which completely correspond to the preset verification dimensions. Each primary indicator has corresponding secondary indicators, including four quantifiable indicators: dimension pass rate, dimension defect rate, percentage of serious defects, and percentage of high-frequency defects. A fixed calculation method is defined for each secondary indicator: the percentage of serious defects is calculated by dividing the number of serious defect items in the corresponding dimension by the total number of defect items in that dimension, multiplied by 100%; the percentage of high-frequency defects is calculated by dividing the number of high-frequency defect items in the corresponding dimension by the total number of defect items in that dimension, multiplied by 100%. The text quality assessment indicator set also reserves an extended entry point for defect category classification, ensuring that all indicators within the indicator set can be directly calculated.

[0075] Next, based on the text quality assessment index set, the target text dataset is classified into defect categories, the text defect dataset is clustered, the frequency of defect type clusters is determined, high-frequency defect types are identified, distribution characteristics are analyzed, and the quality compliance rate is calculated, thereby achieving quality assessment and generating the quality compliance rate. This step will be explained in detail in the following content.

[0076] Subsequently, a qualified threshold for the quality compliance rate is preset. In general scenarios, this qualified threshold is set to 90%. The calculated quality compliance rate is compared with the qualified threshold. If the quality compliance rate does not reach the qualified threshold, the defect root cause tracing process is triggered. Root cause tracing starts with the weakest indicator of the quality compliance rate and traces back to the defect data of the corresponding verification dimension. First, the core weak dimension with the highest defect rate is located. Then, from the defect records of this dimension, high-frequency defect types, the text unit range of defect distribution, the corresponding verification rules triggered, the text type and business scenario information are extracted to analyze the core reasons for the defects. At the same time, the overall distribution characteristics of defects, the frequency of occurrence of high-frequency defects, and the degree of impact on the overall text quality are statistically analyzed. The above content is organized into complete defect analysis data.

[0077] Finally, the basic information of the target text dataset, the details of the verification dimensions, the calculation results of the data pass rate and defect rate, the text quality assessment index set, the details of the quality compliance rate calculation, and the defect analysis data are integrated and arranged in a fixed logical order to generate a complete text data verification report. At the same time, the defect analysis data is formally added to the report as the core analysis module. The final generated text data verification report is associated and stored with the corresponding target text dataset and text verification result label group.

[0078] By completing the above-mentioned sequential steps, a comprehensive text quality assessment and standardized calculation of the quality compliance rate are achieved. Finally, the root causes of defects are traced and a structurally complete and traceable text data verification report is generated. This achieves the effect of conducting a comprehensive, quantitative, and traceable quality assessment of the target text dataset and outputting a standardized and complete text data verification report that can directly guide text optimization.

[0079] In one possible implementation, the quality compliance rate acquisition unit further includes: The system includes the following sub-units: a defect category acquisition sub-unit, used to classify the target text dataset into defect categories according to the text quality assessment index set, and determine multiple defect categories; a defect type cluster acquisition sub-unit, used to extract text defect datasets based on the multiple defect categories, and perform cluster analysis on the text defect datasets to obtain multiple defect type clusters; an occurrence frequency data acquisition sub-unit, used to extract the number of defect records based on the multiple defect type clusters, and determine the occurrence frequency data of the defect type clusters; a high-frequency defect type acquisition sub-unit, used to determine high-frequency defect types when the occurrence frequency data of the defect type clusters exceeds a preset frequency threshold; a distribution feature acquisition sub-unit, used to analyze the target text dataset based on the high-frequency defect types, and determine the distribution features; and a quality compliance rate acquisition sub-unit, used to calculate the quality compliance rate of the target text dataset based on the distribution features, and generate the quality compliance rate.

[0080] Specifically, firstly, the text quality assessment indicator set constructed in the previous steps, and the text verification result label group corresponding to the target text dataset, are retrieved. Using the primary indicators of this indicator set as the core classification basis and the secondary indicators as the subdivision basis, defect categories are divided. The primary dimension corresponds to three core defect categories: format defects, logical defects, and consistency defects. Each core category is further subdivided into subcategories such as dimension defect rate and the proportion of severe defects based on secondary indicators, ensuring that the defect category division is fully aligned with the text quality assessment indicator set. Using keyword matching, each defect record in the text verification result label group is matched to the corresponding core category and subcategory according to the verification dimension and defect features. A unique identifier is assigned to each category, and the total number of defect records and the defect level distribution under each category are recorded. Finally, the defect record classification is completed, determining multiple defect categories corresponding to the quality assessment indicator set.

[0081] Then, defect records are extracted by category. Each defect record retains core fields such as the defect category, defect details, corresponding verification rules, associated text unit features, defect level, text type, and business scenario information. All defect records are standardized in format, unified into structured data, and integrated to form a text defect dataset. Next, the K-means clustering algorithm is used to perform clustering analysis on the text defect dataset. First, four core clustering dimensions are determined: defect verification dimension, trigger rule features, text fragment content features, and defect impact level. The text features of each dimension are quantified into numerical feature vectors using the TF-IDF algorithm. The initial cluster size K is preset to 10, and can be adjusted by increasing K by 1 for every 50 records. The clustering iteration termination condition is set to the convergence of the sum of squared errors within the cluster or the number of iterations reaching 50. After performing the clustering operation, defect records with feature similarity higher than 90% are aggregated into the same set, and each set is a defect type cluster. Each cluster is assigned a unique identifier, and its core defect features, defect category, and defect record details are recorded, ultimately resulting in multiple defect type clusters.

[0082] Next, the total number of valid defect records within each cluster is counted according to the cluster identifier. This value is determined as the frequency data of the corresponding defect type cluster. The statistical period and criteria are completely consistent with the previous steps, using a single batch of text verification tasks as the statistical period. Duplicate defect records are removed to ensure consistent judgment criteria for high-frequency defects. The frequency data is then bound to the cluster identifier, core defect features, and defect category. All defect type clusters are sorted from high to low frequency to complete the collection and organization of frequency data.

[0083] Next, a frequency threshold for defect type clusters is preset. This threshold is completely consistent with the preset frequency threshold in the previous steps. In general scenarios, it is set to 5 times by default, and can be flexibly adjusted within the range of 3 to 10 times depending on the strictness of business verification. The occurrence frequency data of each defect type cluster is compared with the preset frequency threshold one by one. When the frequency data of a defect type cluster is greater than or equal to the preset frequency threshold, its corresponding defect type is marked as a high-frequency defect type. At the same time, the core features, defect category, and defect level distribution details of the high-frequency defect type are extracted. Those that do not reach the threshold are marked as ordinary defect types, thus completing the accurate determination of high-frequency defect types.

[0084] After identifying the high-frequency defect types, their corresponding defect records are retrieved, and multi-dimensional quantitative distribution characteristic analysis is conducted. All analytical dimensions employ percentage-based quantitative statistics. First, the distribution characteristics of defect categories are analyzed, calculating the proportion of each high-frequency defect type within the three core categories of format, logic, and consistency. Next, the distribution characteristics of text units are analyzed, calculating the proportion of high-frequency defects in different text segments and different levels of text units in the target text dataset, thus locating the text range within the defect set. Then, the distribution characteristics of scenarios and types are analyzed, calculating the proportion of high-frequency defects corresponding to different target text types and business scenarios. Finally, the distribution characteristics of defect levels are analyzed, calculating the proportion of severe, moderate, and minor defects among the high-frequency defects. The percentage statistical results from all dimensions are integrated into quantitative distribution characteristic data, and the statistical basis for each characteristic data point is labeled to ensure that the results are verifiable and traceable.

[0085] Next, quality compliance calculations are performed using the text quality assessment indicator set. First, basic scores are assigned to the secondary indicators within this indicator set, with a total basic score of 100 points. This indicator set corresponds to three core categories: format compliance, logical rigor, and content consistency. Each core category has four secondary indicators: dimensional pass rate, dimensional defect rate, percentage of serious defects, and percentage of high-frequency defects. Within a single core category, 10 points are assigned to the dimensional pass rate and 10 points to the dimensional defect rate, and 5 points to the percentage of serious defects and percentage of high-frequency defects. The total score for secondary indicators in a single core category is 30 points, and the total score for secondary indicators across the three core categories is 90 points. The remaining 10 points are the overall performance score. Each indicator is calculated according to its compliance requirements. If any secondary indicator fails to meet the preset threshold, all corresponding basic scores for that indicator are deducted. The overall performance score is assigned at the discretion of the evaluator based on the overall compliance of the three core categories. A perfect score of 10 points is awarded if all secondary indicators in the three core categories meet the requirements. Two points are deducted for each secondary indicator that fails to meet the requirements, until all points are deducted. Finally, the basic quality score of the target text dataset is calculated.

[0086] Then, a weighted deduction coefficient is set based on the distribution characteristics of high-frequency defect types. This coefficient is completely consistent with the preset frequency threshold and defect level classification criteria in the previous steps. In general scenarios, the deduction coefficient for severe high-frequency defects is 1.5, for general defects it is 1.0, and for minor defects it is 0.5. For every 50% increase in the frequency of high-frequency defect type clusters beyond the preset frequency threshold, the corresponding deduction coefficient increases by 0.2, up to a maximum of 2.0. Based on the total proportion of high-frequency defects in the distribution characteristics, the additional deduction score is calculated by multiplying the proportion of high-frequency defects by the weighted deduction coefficient and then by the basic deduction value. The final actual quality score is obtained by subtracting the additional deduction score from the basic quality score. The final actual quality score is divided by the basic full score of 100 and multiplied by 100%, which is the quality compliance rate of the target text dataset. All calculation details are simultaneously retained, including the calculation process of item scoring, basic deduction, weighted coefficient setting, and additional deduction, to ensure that the calculation steps are traceable and verifiable.

[0087] Through the above-mentioned sequential steps, a quality compliance rate that closely matches the actual quality of the target text dataset is generated based on refined defect analysis, thus providing more accurate and quantitative data analysis support for text quality assessment and root cause tracing.

[0088] In summary, the automated text data collection and intelligent verification system provided in this application has the following technical effects: This application identifies and parses multi-source heterogeneous data sources to generate structured and semantic features. After feature mapping and splicing, importance assessment and filtering, the target text dataset is obtained. A dynamic verification rule base is constructed to complete multi-dimensional intelligent verification and generate verification result label groups. Based on the label groups, quality assessment, defect root cause tracing and verification report generation are carried out. Furthermore, the verification rules can be iteratively optimized based on high-frequency defects to realize the whole-link processing of text data collection, verification and evaluation, which greatly improves the accuracy and reliability of text data processing.

[0089] Example 2 like Figure 2 As shown, based on the same inventive concept as Embodiment 1 above, this application provides a method for automated collection and intelligent verification of text data, the method comprising: Text content recognition is performed based on the data source to be processed, an initial text dataset is obtained and parsed to generate structured features and semantic features; automated collection is performed based on the structured features and semantic features to determine the target text dataset; a dynamic verification rule base is constructed to perform multi-dimensional intelligent verification on the target text dataset and generate a text verification result label group; the quality of the target text dataset is evaluated based on the text verification result label group and a text data verification report is generated.

[0090] Furthermore, based on the data source to be processed, text content recognition is performed to obtain an initial text dataset for parsing, generating structured features and semantic features. This application embodiment also includes: The data source to be processed is determined by identifying its type, including structured, semi-structured, and unstructured data sources. Text extraction is performed on the structured data source to obtain first raw text content. Text extraction is then performed on the semi-structured data source to obtain second raw text content. Text extraction is then performed on the unstructured data source to obtain third raw text content. Noise filtering is applied to the first, second, and third raw text contents to construct an initial text dataset. The initial text dataset is then analyzed for layout structure to determine text layout features and hierarchical features. The text layout features and hierarchical features are then integrated to construct structured features. Finally, the initial text dataset is semantically parsed to determine deep text semantic representation data, which is used as semantic features.

[0091] Furthermore, based on the structured features and semantic features, automated data collection is performed to determine the target text dataset. This application embodiment also includes: A feature space is constructed, and the structured features are mapped to the feature space to generate a structured feature vector. The semantic features are also mapped to the feature space to generate a semantic feature vector. The structured feature vector and the semantic feature vector are concatenated to obtain text feature parameters. Based on the text feature parameters, an importance assessment is performed to generate multiple text importance scores. The initial text dataset is traversed and filtered according to the multiple text importance scores to determine key text units. The key text units are then aggregated to generate the target text dataset.

[0092] Furthermore, based on the aforementioned text feature parameters, an importance assessment is performed to generate multiple text importance scores. This application embodiment also includes: The initial text dataset is divided into multiple text units, each containing a continuous text segment. Identifiers are assigned to the multiple text units based on these continuous text segments to determine multiple unit identifiers. Text feature parameters are matched according to these unit identifiers to extract multiple text feature vectors. Deep learning is performed on the multiple text units based on these text feature vectors, and importance is evaluated based on the learning results to obtain the importance scores of the multiple text units.

[0093] Furthermore, the construction process of the dynamic validation rule base, as described in this embodiment, also includes: Historical verification logs are used for pattern recognition to extract a first type of candidate rule; a domain knowledge base is introduced for knowledge extraction to extract a second type of candidate rule; the first and second types of candidate rules are merged and deduplicated to generate an initial verification rule set; the initial verification rule set is traversed for validity verification, and the initial verification rule set is filtered according to the rule validity parameters to determine multiple valid rules; the multiple valid rules are integrated and stored to construct the dynamic verification rule library.

[0094] Furthermore, a dynamic validation rule base is constructed to perform multi-dimensional intelligent validation on the target text dataset, generating text validation result label groups. This application embodiment also includes: Based on the target text dataset, text recognition is performed to determine the target text type; based on the target text dataset, scene recognition is performed to determine business scene information; a dynamic verification rule base is traversed according to the target text type and the business scene information to perform dynamic matching and determine a subset of verification rules; format verification is performed on the target text dataset according to the subset of verification rules to generate a first-dimensional verification result; logical verification is performed on the target text dataset according to the subset of verification rules to generate a second-dimensional verification result; consistency verification is performed on the target text dataset according to the subset of verification rules to generate a third-dimensional verification result; the first-dimensional verification result, the second-dimensional verification result, and the third-dimensional verification result are normalized to obtain verification record parameters; text mapping is performed based on the verification record parameters to construct a text-verification result mapping relationship; the first-dimensional verification result, the second-dimensional verification result, and the third-dimensional verification result are encapsulated according to the text-verification result mapping relationship to generate the text verification result label group.

[0095] Furthermore, after generating the text verification result tag group, the embodiments of this application also include: The text verification result tag group is continuously monitored, and the defect type is determined based on the monitoring results. The occurrence frequency parameter of the defect type is captured, and when the occurrence frequency parameter exceeds a preset frequency threshold, an automatic update mechanism is triggered. Based on the defect type matching text unit features, a first candidate verification rule is constructed through the automatic update mechanism. Effective verification is performed based on the first candidate verification rule to generate a verification result. When the verification result passes, the first candidate verification rule is filled into the initial verification rule set for update and optimization.

[0096] Furthermore, based on the text verification result tag group, the target text dataset is subjected to quality assessment, and a text data verification report is generated. This application embodiment also includes: Multiple verification dimensions are set, and data transmission verification is performed based on the text verification result label group according to the multiple verification dimensions to generate data transmission verification records; data calculation is performed according to the data transmission verification records to obtain the data pass rate and data defect rate; text quality analysis is performed based on the data pass rate and the data defect rate to set a set of text quality assessment indicators; the target text dataset is quality assessed according to the set of text quality assessment indicators to generate a quality compliance rate; root cause tracing is performed based on the quality compliance rate to determine defect analysis data to be added to the text data verification report.

[0097] Furthermore, the target text dataset is evaluated according to the aforementioned text quality assessment index set to generate a quality compliance rate. This embodiment of the application also includes: The target text dataset is categorized into defect categories according to the text quality assessment index set to determine multiple defect categories. A text defect dataset is extracted based on these multiple defect categories, and cluster analysis is performed on the text defect dataset to obtain multiple defect type clusters. The number of defect records is extracted based on the multiple defect type clusters to determine the frequency data of the defect type clusters. When the frequency data of the defect type clusters exceeds a preset frequency threshold, high-frequency defect types are identified. The target text dataset is analyzed based on the high-frequency defect types to determine distribution characteristics. The quality compliance rate of the target text dataset is calculated based on the distribution characteristics to generate the quality compliance rate.

[0098] Through the foregoing detailed description of an automated text data collection and intelligent verification system, those skilled in the art can clearly understand that the automated text data collection and intelligent verification method in this embodiment corresponds to the system disclosed in Embodiment 1, and has corresponding execution steps and technical effects. For relevant details, please refer to the system section description.

[0099] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A system for automated collection and intelligent verification of text-based data, characterized in that, The system includes: The text feature generation module performs text content recognition based on the data source to be processed, obtains the initial text dataset, parses it, and generates structured features and semantic features; The target text dataset acquisition module is used to automatically collect data based on the structured features and semantic features to determine the target text dataset. The text verification result label group generation module is used to construct a dynamic verification rule base to perform multi-dimensional intelligent verification on the target text dataset and generate text verification result label groups. The text data verification report generation module performs a quality assessment on the target text dataset based on the text verification result label group and generates a text data verification report. The system for automatically collecting and determining the target text dataset based on the structured features and semantic features includes: The feature vector generation unit is used to construct a feature space, map the structured features to the feature space, generate a structured feature vector, and map the semantic features to the feature space to generate a semantic feature vector. The text feature parameter acquisition unit is used to concatenate the structural feature vector with the semantic feature vector to obtain text feature parameters. The text importance score generation unit evaluates the importance of the text based on the text feature parameters and generates multiple text importance scores. The key text unit acquisition unit is used to traverse the initial text dataset according to the multiple text importance scores to filter and determine key text units. The target text dataset acquisition unit is used to aggregate the key text units to generate the target text dataset; The system for evaluating the importance of text based on the text feature parameters and generating multiple text importance scores includes: Multiple text units are used to obtain sub-units, which are used to divide the initial text dataset into multiple text units, each of which contains a continuous text segment; Multiple unit identifiers are used to obtain sub-units, and the multiple text units are assigned identifiers based on multiple consecutive text segments to determine multiple unit identifiers; A text feature vector acquisition subunit is used to match the text feature parameters according to the multiple unit identifiers and extract multiple text feature vectors; The text importance score acquisition subunit performs deep learning on the multiple text units based on the multiple text feature vectors, evaluates the importance of the multiple texts based on the learning results, and obtains the multiple text importance scores.

2. The automated text data acquisition and intelligent verification system according to claim 1, characterized in that, The system performs text content recognition based on the data source to be processed, obtains an initial text dataset, parses it, and generates structured features and semantic features. The system includes: The data source type information acquisition unit is used to determine the data source type of the data source to be processed and to determine the data source type information, which includes structured data source, semi-structured data source and unstructured data source. The first original text content acquisition unit is used to perform text extraction operations based on the structured data source to obtain the first original text content; The second original text content acquisition unit is used to perform text extraction operations based on the semi-structured data source to obtain the second original text content; The third original text content acquisition unit is used to perform text extraction operations based on the unstructured data source to obtain the third original text content; An initial text dataset construction unit is used to filter noise from the first original text content, the second original text content, and the third original text content to construct an initial text dataset. The text feature determination unit is used to perform layout structure analysis on the initial text dataset to determine text layout features and hierarchical features. The structured feature construction unit is used to integrate the text layout features and the hierarchical features to construct structured features; The semantic feature acquisition unit is used to perform semantic parsing on the initial text dataset, determine deep text semantic representation data, and use the deep text semantic representation data as semantic features.

3. The automated text data acquisition and intelligent verification system according to claim 1, characterized in that, The system includes the following components: The process of building a dynamic validation rule base. The first type of candidate rule acquisition unit is used to schedule historical verification logs for pattern recognition and extract the first type of candidate rules; The second type of candidate rule acquisition unit is used to introduce the domain knowledge base for knowledge extraction and extract the second type of candidate rules; The initial verification rule set acquisition unit is used to merge and deduplicate the first type of candidate rules and the second type of candidate rules to generate an initial verification rule set; Multiple valid rule determination units are used to traverse the initial verification rule set for validity verification, filter the initial verification rule set according to rule validity parameters, and determine multiple valid rules; The dynamic validation rule base construction unit is used to integrate and store the multiple valid rules to construct the dynamic validation rule base.

4. The automated text data acquisition and intelligent verification system according to claim 3, characterized in that, The system constructs a dynamic validation rule base to perform multi-dimensional intelligent validation on the target text dataset and generates text validation result label groups. The system includes: The target text type acquisition unit performs text recognition based on the target text dataset to determine the target text type; The business scenario information acquisition unit performs scenario recognition based on the target text dataset to determine the business scenario information. The verification rule subset acquisition unit is used to dynamically match the dynamic verification rule library according to the target text type and the business scenario information to determine the verification rule subset. The first dimension verification result acquisition unit is used to perform format verification on the target text dataset according to the subset of verification rules and generate the first dimension verification result. The second-dimensional verification result acquisition unit is used to perform logical verification on the target text dataset according to the subset of verification rules and generate the second-dimensional verification result. The third-dimensional verification result acquisition unit is used to perform consistency verification on the target text dataset according to the subset of verification rules and generate a third-dimensional verification result. The verification record parameter acquisition unit is used to normalize the first dimension verification result, the second dimension verification result, and the third dimension verification result to obtain the verification record parameters. The text verification result tag group generation unit performs text mapping based on the verification record parameters, constructs a text-verification result mapping relationship, and encapsulates the first-dimensional verification result, the second-dimensional verification result, and the third-dimensional verification result according to the text-verification result mapping relationship to generate the text verification result tag group.

5. The automated text data acquisition and intelligent verification system according to claim 4, characterized in that, After generating the text validation result label group, the system includes: The defect type determination subunit is used to continuously monitor the text verification result tag group and determine the defect type based on the monitoring results; An automatic update mechanism execution subunit is used to capture the occurrence frequency parameter of defect type. When the occurrence frequency parameter exceeds a preset frequency threshold, the automatic update mechanism is triggered. The verification result generation subunit is used to construct a first candidate verification rule based on the defect type matching text unit features through the automatic update mechanism, perform effective verification based on the first candidate verification rule, and generate a verification result. The initial verification rule set update subunit is used to fill the first candidate verification rule into the initial verification rule set for update and optimization when the verification result passes.

6. The automated text data acquisition and intelligent verification system according to claim 1, characterized in that, The system performs a quality assessment of the target text dataset based on the text validation result label group and generates a text data validation report. The system includes: The data transmission verification record acquisition unit is used to set multiple verification dimensions, perform data transmission verification based on the text verification result tag group according to the multiple verification dimensions, and generate data transmission verification records. The data parameter acquisition unit is used to perform data calculations according to the data transmission verification record to obtain the data pass rate and data defect rate. The text quality assessment index set construction unit performs text quality analysis based on the data pass rate and the data defect rate, and sets a text quality assessment index set. The quality compliance rate acquisition unit is used to perform quality assessment on the target text dataset according to the text quality assessment index set and generate a quality compliance rate. The defect analysis data acquisition unit performs root cause tracing based on the quality compliance rate and determines that the defect analysis data will be added to the text data verification report.

7. The automated text data acquisition and intelligent verification system according to claim 6, characterized in that, The system performs a quality assessment on the target text dataset according to the aforementioned text quality assessment index set, and generates a quality compliance rate. The system includes: The defect category acquisition subunit is used to classify the target text dataset into defect categories according to the text quality assessment index set, and determine multiple defect categories. The defect type cluster acquisition subunit extracts a text defect dataset based on the multiple defect categories, performs cluster analysis on the text defect dataset, and obtains multiple defect type clusters. The frequency data acquisition subunit is used to extract the number of defect records based on the multiple defect type clusters and determine the frequency data of the defect type clusters. The high-frequency defect type acquisition subunit is used to determine the high-frequency defect type when the occurrence frequency data of the defect type cluster exceeds a preset frequency threshold. The distribution feature acquisition subunit analyzes the target text dataset based on the high-frequency defect types to determine the distribution features; The quality compliance rate acquisition subunit is used to calculate the quality compliance of the target text dataset based on the distribution characteristics and generate the quality compliance rate.

8. A method for automated collection and intelligent verification of text-based data, characterized in that, The method is implemented by the automated text data acquisition and intelligent verification system according to any one of claims 1-7, and the method includes: Text content recognition is performed based on the data source to be processed, the initial text dataset is parsed, and structured features and semantic features are generated. The target text dataset is determined by automatically collecting data based on the structured features and semantic features. A dynamic validation rule base is constructed to perform multi-dimensional intelligent validation on the target text dataset, generating a text validation result label group. Based on the text verification result label group, the quality of the target text dataset is evaluated, and a text data verification report is generated. Construct a feature space, map the structured features to the feature space to generate a structured feature vector, and map the semantic features to the feature space to generate a semantic feature vector; The structural feature vector and the semantic feature vector are concatenated to obtain the text feature parameters; Based on the text feature parameters, an importance assessment is performed to generate multiple text importance scores; The initial text dataset is filtered by iterating through the multiple text importance scores to determine the key text units; The key text units are aggregated to generate the target text dataset; The initial text dataset is divided into multiple text units, each of which contains a continuous text segment; Based on multiple consecutive text segments, the multiple text units are identified and assigned identifiers to determine multiple unit identifiers; The text feature parameters are matched according to the multiple unit identifiers to extract multiple text feature vectors; Deep learning is performed on the multiple text units based on the multiple text feature vectors, and importance evaluation is performed based on the learning results to obtain the importance scores of the multiple texts.

Images