Resource consumption intelligent supervision system and method for document data quality detection

Abstract

The invention discloses an intelligent resource consumption supervision system and method for document data quality detection, and relates to the technical field of document detection, metadata information and structural feature information of documents for data quality detection are extracted, and an information document set is constructed; carrying out normalization processing on the metadata information and the structural feature information of the documents, and grouping the documents; calculating the structural complexity of the documents in the groups based on the groups, selecting representative documents according to the structural complexity, and performing structural analysis on the representative documents; constructing a cache template used for detecting the corresponding grouped documents according to the information representing the documents; performing document data quality detection on the non-representative documents in the groups based on the cache template; and when a newly added document is added into the group, constructing and triggering a cache updating mechanism, updating the cache template, and completing data quality detection of the document.

Description

Technical Field

[0001] The present invention relates to the technical field of document detection, and in particular to a resource consumption intelligent monitoring system and method for document data quality detection. Background Art

[0002] In modern document processing and quality inspection, with the increase in document types and content complexity, traditional document quality inspection methods usually rely on checking each part of the document one by one, including metadata verification, text content consistency checking, format checking, and structure analysis. Although these methods can ensure document quality, when the document volume is large or the content is complex, the processing will consume a lot of computing resources and time. In particular, when multiple rounds of inspection are required for multiple documents, efficiency and performance bottlenecks are particularly obvious. Current document detection methods often suffer from high resource consumption and low processing efficiency. In particular, traditional detection methods rely on re-parsing and analyzing the entire document each time. These methods typically use rule engines and fixed algorithms for full-text processing. Every time a document changes, a complete analysis and detection of the document is required, resulting in a waste of computing resources. In addition, resource management in traditional methods is often based on fixed rule allocation, and resources cannot be dynamically adjusted according to the complexity of the document or the processing stage, thus failing to effectively avoid performance issues such as high load and long waiting times. Therefore, there is an urgent need for an intelligent resource consumption supervision system and method for document data quality detection to solve the above problems. Summary of the Invention

[0003] The purpose of the present invention is to provide a method to solve the problems raised in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a resource consumption intelligent supervision method for document data quality detection, the method comprising the following steps: S100 extracts metadata and structural feature information from documents undergoing data quality inspection and constructs an information document collection; Furthermore, the specific steps for extracting metadata information and structural feature information of the document are: S101 collects documents for quality inspection and constructs an original document set D={d1,d2,d3,...,dn}, where d1,d2,d3,... and dn represent the first, second, third,... and nth documents respectively. For the xth document, x∈[1,n], the metadata information of the document is extracted by the document parser and recorded as Yx. Metadata is the basic information of the document, including but not limited to the following: document type, document size, number of document pages, creation time / modification time and author; the metadata extraction result is represented as a set Xy={Yx1,Yx2,Yx3,...,Yxm}, where Yx1,Yx2,Yx3,... and Yxm represent The first, second, third, ..., and mth metadata information of the xth document are extracted, and the xth document structural feature information Tx is extracted by parsing the document content. Document structural features refer to the layout, relationship, and semantic information of each element in the document. Extraction methods include but are not limited to: natural language processing (NLP), regular expressions, layout analysis, optical character recognition, and machine learning and deep learning. The extracted structural features are represented as a set Xt={Tx1,Tx2,Tx3,...,Txb}, where Tx1,Tx2,Tx3,... and Txb represent the first, second, third, ..., and bth structural feature information of the xth document, respectively. S102 combines the metadata information and structural feature information of the x-th document to form an information description of the x-th document, expressed as dx*={Xy,Xt}, and constructs a document set D^={d1*,d2*,d3*,...,dn*} represented by the document information, where d1*, d2*, d3*,... and dn* respectively represent the first, second, third,... and n-th documents containing the document information.

[0005] S200 normalizes the metadata information and structural feature information of the documents and groups the documents; Furthermore, the specific steps for grouping documents are: S201: normalizing the structural feature information and the numerical information in the metadata of the x-th document using a minimum-maximum method. The numerical information in the structural features includes the number of paragraphs, the number of charts, the depth of the title hierarchy, the number of table rows and columns, and the numerical information in the metadata includes the document size, the number of document pages, and the creation and modification timestamps. The categorical information in the structural feature information and the metadata includes the icon type, the title style, and the paragraph alignment, and the categorical information in the metadata includes the document type, the department described by the author, and the document classification level. The categorical information is converted into a one-hot encoding to obtain processed structural feature information Tx' and metadata information Yx'. The processed structural feature information and metadata information of the x-th document are merged to construct a final feature set dx'={Yx1', Yx2', Yx3', ..., Yxm', Tx1', Tx2', Tx3', ..., Txb'} of the x-th document. S202 calculates the similarity Sxz between the x-th document and the z-th document based on the data in the processed feature set, where z∈[1,n] and z≠x. The calculation formula is: , Where k represents the data index of the metadata in the final feature set, and j represents the data index of the structural feature in the final feature set; S203 calculates the similarity between any two documents for all documents in the set D^ and performs statistical analysis on the similarity, calculates the mean μS and standard deviation σS of the similarity, and calculates the initial similarity threshold θmin based on the mean and standard deviation. The calculation formula is: , The similarity between the xth and zth documents is compared with the threshold. When Sxz≥θmin, the xth document is taken as the core document of this group. The similarity between the remaining documents in the set D^ except the xth and zth documents and the xth document is compared with the threshold. Documents larger than the threshold are divided into the group containing the xth document, and the remaining documents are not grouped temporarily. When Sxz<θmin, classification is not performed temporarily. For documents that are not classified, an adjustment factor λ is introduced. The value of λ is dynamically adjusted according to the real-time classification performance, such as accuracy, recall rate, etc., and the reduction ratio of unclassified documents. When the classification effect is good and the reduction of unclassified documents is significant, λ can be appropriately increased, and the threshold is increased accordingly to avoid misclassification due to loose classification standards when the reduction of unclassified documents is significant, thereby ensuring the classification quality. When the classification effect deteriorates or the number of unclassified documents decreases, the value of λ is adjusted according to the real-time classification performance, such as accuracy, recall rate, etc., and the reduction ratio of unclassified documents. When the classification effect is good and the number of unclassified documents decreases significantly, λ can be appropriately increased, and the threshold is increased accordingly. This is to avoid misclassification due to loose classification standards when the number of unclassified documents decreases significantly, thereby ensuring the classification quality. When the speed is slow, reduce λ, and the corresponding threshold will decrease accordingly. At this time, the similarity of more documents can meet the threshold requirement, thereby reducing the number of unclassified documents and improving classification efficiency. Adjust the threshold θmin to obtain a new threshold θmin1. The calculation formula of θmin1 is: θmin1=μS+λσS. According to the threshold θmin1, the unclassified documents are compared with the threshold and classified again. The maximum number of adjustments is set to Tmax. When the maximum number of adjustments is reached, the adjustment is stopped. If the adjustment number reaches Tmax and is still not classified, the document is classified into a separate group Gd+1. Finally, the n documents are divided into several groups to obtain a group set G={G1,G2,G3,...,Gd,Gd+1}, where G1, G2, G3,...,Gd and Gd+1 represent the first, second, third,...,dth and separate groups respectively. S300 calculates the structural complexity of the documents in the group based on the grouping, selects a representative document according to the structural complexity, and performs structural analysis on the representative document; Furthermore, the specific steps for structural analysis of the representative document are as follows: S301 calculates the structural complexity Ci of the i-th document in the e-th group, e∈[1,d], i represents any document in the e-th group, and the calculation formula is: , Where ω1, ω2, ω3, ..., and ωb are the weights of each structural feature in the i-th document on the document complexity. The weight selection should be based on the actual impact of the document structural features and specific needs. Expert knowledge, statistical methods, machine learning models, and other methods can be used to assign appropriate weights to each feature. One method is now described: by collecting several historical document detection data, recording the computing resource consumption corresponding to each structural feature, such as parsing time and memory usage, and calculating the resource consumption ratio of a single structural feature. The calculation formula is: ,in Indicates the resource consumption ratio of any single structural feature, Tre is the average resource consumption of any single structural feature, Represents the total resource consumption of all structural features, and the weight is determined according to the resource consumption ratio; select the document with the largest structural complexity from the e-th group as the representative document dc of the group; S302 analyzes the overall structure of the representative document in the e-th group through document element identification, title hierarchy analysis, chapter division, and chart positioning to obtain a structural analysis Tc, including document element identification (cover, table of contents, chapters, paragraphs, charts, tables); title hierarchy analysis (based on font style and alignment); chapter division (based on tags such as "Chapter X"); and chart positioning (recording location and type). Then, corresponding title detection rules, paragraph formatting rules, table detection rules, and chart detection rules are formulated for each structural element of the representative document to construct a detection rule set Rc. A set of specific detection rules is defined for each document structural unit (such as a title, paragraph, table, and chart). These rules include how to identify the title, paragraph format, table structure, and chart format. Then, a title content template, paragraph content template, table content template, and chart content template are defined for each structural unit in the representative document. Furthermore, title format constraints, paragraph format constraints, table format constraints, and chart format constraints are defined to obtain a content constraint template Pc. S400 constructs a cache template for corresponding grouped document detection based on the information representing the document; Furthermore, the specific steps for constructing a cache template for corresponding group document detection are: S401 divides the representative document in the e-th group into several structural units Lj according to structural analysis. The document is divided into multiple structural units, which are various parts of the document, such as: title: first-level title, second-level title, etc., paragraph: each text paragraph, table: the independent part of each table, chart: the independent part of each chart, these structural units are identified based on the structural analysis results of the document through title hierarchy, chapter division, etc., and constructs a structural unit set L={Lj1,Lj2,Lj3,...,Ljw}, where Lj1,Lj2,Lj3,... and Ljw represent the first, second, third,... and w-th structural units respectively. For the a-th structural unit in the set L, a∈[1,w], it contains the corresponding detection rule set Rca and content constraint template Pca; S402 stores the information of the w structural units as a cache template Md, expressed as Md={(Lj1, Tc1, Rc1, Pc1), (Lj2, Tc2, Rc2, Pc2), (Lj3, Tc3, Rc3, Pc3), ..., (Ljw, Tcw, Rcw, Pcw)}, where (Lj1, Tc1, Rc1, Pc1), (Lj2, Tc2, Rc2, Pc2), (Lj3, Tc3, Rc3, Pc3), ... and (Ljw, Tcw, Rcw, Pcw) represent the first, second, third, ... and wth structural units and the cache template contents corresponding to the structural units, respectively, and uses the cache template Md as the cache template corresponding to the eth group; By storing the structural unit information (parsing results, detection rules, and constraint templates) in a cache template, the system avoids the resource-intensive parsing and rule building required for each detection. This effectively reduces the system's computational load and memory consumption during document detection. Based on the complexity and type of the document, the information in the cache template can be flexibly used without having to perform a full calculation each time, thus reducing unnecessary resource consumption. S500 performs document data quality detection on non-representative documents in the group based on the cache template; Furthermore, the specific steps for performing document data quality detection on non-representative documents in the group are as follows: S501: For other non-representative documents in the e-th group, the non-representative documents are divided into a number of structural units based on structural analysis. For each structural unit in the non-representative document, if there is a structural unit in the non-representative document that corresponds to the structural unit in the cache template, the detection rule set and content constraint template of the corresponding structural unit in the cache template are directly reused; S502 When a structural unit in a non-representative document does not exist in the cache template, the corresponding structural unit is marked as a new structural unit, and a temporary detection rule set and a content constraint template are generated as a temporary cache and added to the cache template. A temporary cache survival time threshold ξ is set. When the temporary cache detection is not triggered within the threshold time, the temporary cache is deleted. It should be further explained that a series of constraints are introduced in the generation of the temporary detection rule set and the survival time threshold ξ. First, the frequency of rule set generation is limited to avoid creating new caches too frequently, and the survival time of the cache is evaluated based on the effectiveness of the cache. If the temporary cache does not trigger detection, it is automatically deleted when the set threshold is exceeded. Second, the rule set is prevented from being too complex or repeated. By merging similar rule sets and limiting the number of rule set contents, efficient use of system resources is ensured. In addition, a dynamic adjustment of the cache quantity, frequency and validity check mechanism is adopted to ensure the stability of the temporary cache system and effectively prevent resource waste. Dynamic cache management and an adaptive rule generation mechanism make document processing more flexible and efficient. This not only improves document processing efficiency and reduces computing and storage resource consumption, but also enhances the system's adaptability to new document structures and formats. By intelligently managing cache lifetimes and update rules, the system can continuously optimize and improve detection accuracy and efficiency, ensuring high-quality document processing and management. When a new document is added to the group, S600 builds and triggers a cache update mechanism, updates the cache template, and completes data quality testing of the document; Furthermore, the specific steps to build and trigger the cache update mechanism are: S601: For the newly added document requiring data quality inspection, calculate the similarity with the representative documents in each group, select the group to which the representative document with the maximum similarity belongs, and add the newly added document to the group; S602: For the e-th group, when a new document is added, a threshold value for the number of new documents in the group is set to Y. The value of Y can be adjusted according to the system load. When the system load is high, the value of Y can be appropriately increased to reduce frequent organizational structure adjustments to avoid excessive resource consumption. When the system load is low, Y can be set to a smaller value. When the number of new documents added to the e-th group exceeds the threshold, the structural complexity of the Y new documents in the e-th group is calculated and compared with the structural complexity of the representative document dc in the group. When the structural complexity of the new document is less than or equal to the structural complexity of the representative document dc, the cache template constructed using the representative document is used to perform data quality testing on the new document. When the structural complexity of the new document is greater than the structural complexity of the representative document dc, the new document is used as a new representative document dc*, the new representative document is divided into structural units, a new cache template Md* is constructed as the cache template of the e-th group, the cache template Md is replaced, and data quality testing is performed on the new document. When performing data quality detection through cache templates, cache failure may occur. Cache failure is caused by a variety of reasons, including changes in document content or structure, format updates, detection rule modifications, system environment changes, cache space limitations, time expiration, similarity changes, user operation changes, and processing of multiple versions or multiple sources of documents. To solve these problems, a dynamic cache management mechanism can be adopted, combined with document version and structure analysis, to update cache content in real time; set cache expiration time, regularly clean up invalid caches; and intelligently monitor detection rule changes and document updates. By dynamically choosing whether to update cache templates based on the structural complexity of newly added documents, we ensure efficient document processing while ensuring the system's flexible adaptability to changes in document structure complexity. This effectively optimizes the use of computing and storage resources, improves the accuracy and consistency of document quality testing, and maintains high document processing efficiency. In order to better implement the above method, a resource consumption intelligent supervision system for document data quality detection is proposed. The system includes a feature extraction module, a document grouping module, a parsing module, a cache template construction module, a document detection module and a cache update module. The feature extraction module is used to extract metadata information and structural feature information of the document for data quality detection; The document grouping module is used to normalize the metadata information and structural feature information of the documents and group the documents; The parsing module is used to calculate the structural complexity of the documents in the group, select representative documents based on the structural complexity, and perform structural analysis on the representative documents; The parsing module includes a complexity calculation unit and a representative document recognition unit; The complexity calculation unit is used to calculate the structural complexity of the documents in the group; The representative document identification unit is used to identify the document with the highest structural complexity in the group as the representative document; The cache template construction module is used to construct a cache template for corresponding group document detection based on information representing the document; The document detection module is used to perform document data quality detection on non-representative documents in the group based on the cache template; The cache update module is used to build and trigger the cache update mechanism when new documents are added to the group.

[0006] Compared with the prior art, the present invention has the following beneficial effects: 1. By analyzing the structural features of documents and storing cache templates, existing templates can be reused during the detection process, avoiding the high overhead of re-analyzing the document structure every time the test is performed, greatly improving processing efficiency. When the complexity of newly added documents changes, the system can automatically select the appropriate cache template for processing based on the structural complexity of the document. This enables the system to efficiently process a large number of documents of different complexities and improve the overall processing speed. 2. By dynamically adjusting the cache template used based on the structural complexity of the document, the system can flexibly respond to documents of different structures to ensure adaptability to different document structures. For each document, especially newly added documents, the system can automatically generate or adjust the cache template based on its structural complexity, avoiding the lack of adaptability caused by the application of static templates and ensuring the flexibility of document detection; 3. As the number of documents increases and the types of documents become more diverse, the system can automatically adapt to the complexity and structure of newly added documents. By dynamically updating cache templates, it ensures effective processing of the growing and changing document library. By analyzing newly added documents, it can gradually optimize the document processing process. As more documents are added, the system can continuously improve efficiency and accuracy and has the ability to self-optimize. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a schematic diagram of the structure of the resource consumption intelligent monitoring system for document data quality detection according to the present invention; Figure 2 Schematic diagram of the process of the intelligent resource consumption supervision method for document data quality detection of the present invention; Figure 3 Schematic diagram of an embodiment of the intelligent resource consumption supervision method for document data quality detection of the present invention. DETAILED DESCRIPTION

[0008] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0009] Example: Figure 1-Figure 3 As shown, the present invention provides a technical solution, a resource consumption intelligent supervision method for document data quality detection, the method comprising the following steps: S100 extracts metadata and structural feature information from documents undergoing data quality inspection and constructs an information document collection; Furthermore, the specific steps for extracting metadata information and structural feature information of the document are: S101 collects documents for quality inspection and constructs an original document set D={d1,d2,d3,...,dn}, where d1,d2,d3,... and dn represent the first, second, third,... and nth documents respectively. For the xth document, x∈[1,n], the metadata information of the document is extracted by the document parser and recorded as Yx. Metadata is the basic information of the document, including but not limited to the following: document type, document size, number of document pages, creation time / modification time and author; the metadata extraction result is represented as a set Xy={Yx1,Yx2,Yx3,...,Yxm}, where Yx1,Yx2,Yx3,... and Yxm represent The first, second, third, ..., and mth metadata information of the xth document are extracted, and the xth document structural feature information Tx is extracted by parsing the document content. Document structural features refer to the layout, relationship, and semantic information of each element in the document. Extraction methods include but are not limited to: natural language processing (NLP), regular expressions, layout analysis, optical character recognition, and machine learning and deep learning. The extracted structural features are represented as a set Xt={Tx1,Tx2,Tx3,...,Txb}, where Tx1,Tx2,Tx3,... and Txb represent the first, second, third, ..., and bth structural feature information of the xth document, respectively. S102 combines the metadata information and structural feature information of the x-th document to form an information description of the x-th document, expressed as dx*={Xy,Xt}, and constructs a document set D^={d1*,d2*,d3*,...,dn*} represented by the document information, where d1*, d2*, d3*,... and dn* respectively represent the first, second, third,... and n-th documents containing the document information.

[0010] S200 normalizes the metadata information and structural feature information of the documents and groups the documents; Furthermore, the specific steps for grouping documents are: S201: normalizing the structural feature information and the numerical information in the metadata of the x-th document using a minimum-maximum method. The numerical information in the structural features includes the number of paragraphs, the number of charts, the depth of the title hierarchy, the number of table rows and columns, and the numerical information in the metadata includes the document size, the number of document pages, and the creation and modification timestamps. The categorical information in the structural feature information and the metadata includes the icon type, the title style, and the paragraph alignment, and the categorical information in the metadata includes the document type, the department described by the author, and the document classification level. The categorical information is converted into a one-hot encoding to obtain processed structural feature information Tx' and metadata information Yx'. The processed structural feature information and metadata information of the x-th document are merged to construct a final feature set dx'={Yx1', Yx2', Yx3', ..., Yxm', Tx1', Tx2', Tx3', ..., Txb'} of the x-th document. S202 calculates the similarity Sxz between the x-th document and the z-th document based on the data in the processed feature set, where z∈[1,n] and z≠x. The calculation formula is: , Where k represents the data index of the metadata in the final feature set, and j represents the data index of the structural feature in the final feature set; S203 calculates the similarity between any two documents for all documents in the set D^ and performs statistical analysis on the similarity, calculates the mean μS and standard deviation σS of the similarity, and calculates the initial similarity threshold θmin based on the mean and standard deviation. The calculation formula is: , The similarity between the xth and zth documents is compared with the threshold. When Sxz≥θmin, the xth document is taken as the core document of this group. The similarity between the remaining documents in the set D^ except the xth and zth documents and the xth document is compared with the threshold. Documents larger than the threshold are divided into the group containing the xth document, and the remaining documents are not grouped for the time being. When Sxz<θmin, classification is not performed for the time being. For documents that are not classified, by introducing the adjustment factor λ, λ∈[0,1], the classification performance indicators accuracy P, recall rate R and the reduction ratio of unclassified documents ΔU (ΔU=(unclassified number before adjustment-unclassified number after adjustment) / unclassified number before adjustment, accuracy P refers to the number of unclassified documents after grouping) are monitored in real time. The ratio of the number of documents correctly classified into a group to the actual number of all documents in the group is used to measure the accuracy of the grouping (to avoid classifying irrelevant documents into the group). The recall rate R refers to the ratio of the number of documents correctly classified into a group after grouping to the total number of documents that should be classified into the group. It is used to measure the completeness of the grouping (to avoid missing documents that should be classified into the group). The judgment threshold is set. For example, P ≥ 85% and R ≥ 80% indicate good classification effect, and ΔU ≥ 30% indicates significant reduction. When "P ≥ 85% and R ≥ 80%" and "ΔU ≥ 30%" are met at the same time, it means that the current classification standard is effective and the number of unclassified documents is significantly reduced. The increase of λ by 0.1 (upper limit 1) is based on the significant reduction of unclassified documents and the maintenance of strict classification standards. , to ensure the classification quality; when "P < 70% or R < 65%" (poor effect) or "ΔU < 10%" (slow reduction), it indicates that under the current classification standard, there are problems such as a high threshold leading to a large number of misclassifications or difficulty in reducing unclassified documents. λ is reduced by 0.2 (lower limit 0), and the similarity of more documents can meet the threshold requirements, thereby relaxing the classification standard to reduce misclassifications and promote the division of unclassified documents; in other intermediate states (P is between 70%-85%, recall rate R is between 65%-80%, and the reduction ratio of unclassified documents ΔU is between 10%-30%), λ is fine-tuned by ±0.05 to balance the classification quality and the speed of unclassified document reduction, so that λ can be dynamically adapted to the real-time classification effect until the maximum adjustment is reached. times Tmax; adjust the threshold θmin to obtain a new threshold θmin1, the calculation formula of θmin1 is: θmin1=μS+λσS, according to the threshold θmin1, the unclassified documents are compared with the threshold and classified again, and the maximum adjustment number is set to Tmax. When the maximum adjustment number is reached, the adjustment is stopped. If the adjustment number reaches Tmax and the document is still not classified, the document is classified into a separate group Gd+1. Finally, the n documents are divided into several groups to obtain a group set G={G1,G2,G3,...,Gd,Gd+1}, where G1, G2, G3,...,Gd and Gd+1 represent the first, second, third,...th, dth and separate groups respectively; S300 calculates the structural complexity of the documents in the group based on the grouping, selects a representative document according to the structural complexity, and performs structural analysis on the representative document; Furthermore, the specific steps for structural analysis of the representative document are as follows: S301 calculates the structural complexity Ci of the i-th document in the e-th group, e∈[1,d], i represents any document in the e-th group, and the calculation formula is: , Where ω1, ω2, ω3, ..., and ωb are the weights of each structural feature in the i-th document on the document complexity. The weight selection should be based on the actual impact of the document structural features and specific needs. Expert knowledge, statistical methods, machine learning models, and other methods can be used to assign appropriate weights to each feature. One method is now described: by collecting several historical document detection data, recording the computing resource consumption corresponding to each structural feature, such as parsing time and memory usage, and calculating the resource consumption ratio of a single structural feature. The calculation formula is: ,in Indicates the resource consumption ratio of any single structural feature, Tre is the average resource consumption of any single structural feature, Represents the total resource consumption of all structural features, and the weight is determined according to the proportion of resource consumption; Assume that the structural features of the i-th document include: Ti1: number of title levels, Ti2: number of paragraphs, Ti3: number of tables and charts, Ti4: number of pages of the document. During the historical detection process, the resource consumption is: parsing time (Ti1: 15, Ti2: 15, Ti3: 10, Ti4: 5), unit is S, memory usage (Ti1: 50, Ti2: 70, Ti3: 30, Ti4: 20), unit is MB. The maximum-minimum normalization method is used to standardize the data. Then, according to the above method, the resource consumption ratio corresponding to each structural unit is obtained. The weight of each structural feature can be determined as its ratio value through the calculated resource consumption ratio. Then, the document with the largest structural complexity is selected from the e-th group as the representative document dc of the group. S302 analyzes the overall structure of the representative document in the e-th group through document element recognition, title hierarchy analysis, chapter division and chart positioning to obtain a structural analysis Tc, document element recognition (cover, table of contents, chapters, paragraphs, charts, tables); title hierarchy analysis (based on font style, alignment); chapter division (based on "Chapter X" and other tags); chart positioning (record location and type), and then formulates corresponding title detection rules, paragraph format rules, table detection rules and chart detection rules for each structural element of the representative document, constructs a detection rule set Rc, and defines a set of specific detection rules for each document structural unit (such as title, paragraph, table, chart), which include how to identify the title, paragraph format, table structure, and chart format, etc., and then defines a title content template, paragraph content template, table content template and chart content template for each structural unit in the representative document, and defines title format constraints, paragraph format constraints, table Format constraints and chart format constraints are used to obtain the content constraint template Pc. Content templates and format constraints are defined for each structural unit (such as titles, paragraphs, tables, charts, etc.). For example, for titles, the content that should be considered a legal title is defined, and its font, size, alignment, etc. are constrained. For paragraphs, the paragraph layout and format are defined. Title content templates include "Chapter X: Chapter Title"; paragraph content templates include the basic sentence structure and format requirements that a paragraph should contain; table content templates include constraints such as the columns and rows, fonts, alignment, etc. that a table should contain; and chart content templates include chart types, labels, and how to present data. S400 constructs a cache template for corresponding grouped document detection based on the information representing the document; Furthermore, the specific steps for constructing a cache template for corresponding group document detection are: S401 divides the representative document in the e-th group into several structural units Lj according to structural analysis. The document is divided into multiple structural units, which are various parts of the document, such as: title: first-level title, second-level title, etc., paragraph: each text paragraph, table: the independent part of each table, chart: the independent part of each chart, these structural units are identified based on the structural analysis results of the document through title hierarchy, chapter division, etc., and constructs a structural unit set L={Lj1,Lj2,Lj3,...,Ljw}, where Lj1,Lj2,Lj3,... and Ljw represent the first, second, third,... and w-th structural units respectively. For the a-th structural unit, a∈[1,w], it contains the corresponding detection rule set Rca and content constraint template Pca; S402 stores the information of the w structural units as a cache template Md, expressed as Md={(Lj1, Tc1, Rc1, Pc1), (Lj2, Tc2, Rc2, Pc2), (Lj3, Tc3, Rc3, Pc3), ..., (Ljw, Tcw, Rcw, Pcw)}, where (Lj1, Tc1, Rc1, Pc1), (Lj2, Tc2, Rc2, Pc2), (Lj3, Tc3, Rc3, Pc3), ... and (Ljw, Tcw, Rcw, Pcw) represent the first, second, third, ... and wth structural units and the cache template contents corresponding to the structural units, respectively, and uses the cache template Md as the cache template corresponding to the eth group; By storing the structural unit information (parsing results, detection rules, and constraint templates) in a cache template, the system avoids the resource-intensive parsing and rule building required for each detection. This effectively reduces the system's computational load and memory consumption during document detection. Based on the complexity and type of the document, the information in the cache template can be flexibly used without having to perform a full calculation each time, thus reducing unnecessary resource consumption. S500 performs document data quality detection on non-representative documents in the group based on the cache template; Furthermore, the specific steps for performing document data quality detection on non-representative documents in the group are as follows: S501: For other non-representative documents in the e-th group, the non-representative documents are divided into a number of structural units based on structural analysis. For each structural unit in the non-representative document, if there is a structural unit in the non-representative document that corresponds to the structural unit in the cache template, the detection rule set and content constraint template of the corresponding structural unit in the cache template are directly reused; S502 When a structural unit in a non-representative document does not exist in the cache template, the corresponding structural unit is marked as a new structural unit, and a temporary detection rule set and a content constraint template are generated as a temporary cache and added to the cache template. A temporary cache survival time threshold ξ is set. When the temporary cache detection is not triggered within the threshold time, the temporary cache is deleted. It should be further explained that a series of constraints are introduced in the generation of the temporary detection rule set and the survival time threshold ξ. First, the frequency of rule set generation is limited to avoid creating new caches too frequently, and the survival time of the cache is evaluated based on the effectiveness of the cache. If the temporary cache does not trigger detection, it is automatically deleted when the set threshold is exceeded. Second, the rule set is prevented from being too complex or repeated. By merging similar rule sets and limiting the number of rule set contents, efficient use of system resources is ensured. In addition, a dynamic adjustment of the cache quantity, frequency and validity check mechanism is adopted to ensure the stability of the temporary cache system and effectively prevent resource waste. Dynamic cache management and an adaptive rule generation mechanism make document processing more flexible and efficient. This not only improves document processing efficiency and reduces computing and storage resource consumption, but also enhances the system's adaptability to new document structures and formats. By intelligently managing cache lifetimes and update rules, the system can continuously optimize and improve detection accuracy and efficiency, ensuring high-quality document processing and management. When a new document is added to the group, S600 builds and triggers a cache update mechanism, updates the cache template, and completes data quality testing of the document; Furthermore, the specific steps to build and trigger the cache update mechanism are: S601: For the newly added document requiring data quality inspection, calculate the similarity with the representative documents in each group, select the group to which the representative document with the maximum similarity belongs, and add the newly added document to the group; S602: For the e-th group, when a new document is added, a threshold value for the number of new documents in the group is set to Y. The value of Y can be adjusted according to the system load. When the system load is high, the value of Y can be appropriately increased, for example, set to 15% of the number of documents in the group, to reduce frequent group structure adjustments and avoid excessive resource consumption. When the system load is low, Y can be set to a smaller value, such as 5% of the number of documents in the group. When the number of new documents added to the e-th group exceeds the threshold, the structural complexity of the Y new documents in the e-th group is calculated and compared with the structural complexity of the representative document dc in the group. When the structural complexity of the new document is less than or equal to the structural complexity of the representative document dc, the cache template constructed by the representative document is used to perform data quality testing on the new document. When the structural complexity of the new document is greater than the structural complexity of the representative document dc, the new document is used as a new representative document dc*, the new representative document is divided into structural units, a new cache template Md* is constructed as the cache template of the e-th group, the cache template Md is replaced, and data quality testing is performed on the new document. When performing data quality detection through cache templates, cache failure may occur. Cache failure is caused by a variety of reasons, including changes in document content or structure, format updates, detection rule modifications, system environment changes, cache space limitations, time expiration, similarity changes, user operation changes, and processing of multiple versions or multiple sources of documents. To solve these problems, a dynamic cache management mechanism can be adopted, combined with document version and structure analysis, to update cache content in real time; set cache expiration time, regularly clean up invalid caches; and intelligently monitor detection rule changes and document updates. By dynamically choosing whether to update cache templates based on the structural complexity of newly added documents, we ensure efficient document processing while ensuring the system's flexible adaptability to changes in document structure complexity. This effectively optimizes the use of computing and storage resources, improves the accuracy and consistency of document quality testing, and maintains high document processing efficiency. In order to better implement the above method, a resource consumption intelligent supervision system for document data quality detection is proposed. The system includes a feature extraction module, a document grouping module, a parsing module, a cache template construction module, a document detection module and a cache update module. The feature extraction module is used to extract metadata information and structural feature information of the document for data quality detection; The document grouping module is used to normalize the metadata information and structural feature information of the documents and group the documents; The parsing module is used to calculate the structural complexity of the documents in the group, select representative documents based on the structural complexity, and perform structural analysis on the representative documents; The parsing module includes a complexity calculation unit and a representative document recognition unit; The complexity calculation unit is used to calculate the structural complexity of the documents in the group; The representative document identification unit is used to identify the document with the highest structural complexity in the group as the representative document; The cache template construction module is used to construct a cache template for corresponding group document detection based on information representing the document; The document detection module is used to perform document data quality detection on non-representative documents in the group based on the cache template; The cache update module is used to build and trigger the cache update mechanism when new documents are added to the group.

[0011] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. An intelligent resource consumption monitoring method for document data quality detection, characterized by: S100 extracts metadata and structural feature information from documents undergoing data quality inspection and constructs an information document collection; S200 normalizes the metadata information and structural feature information of the documents and groups the documents; S300 calculates the structural complexity of the documents in the group based on the grouping, selects a representative document according to the structural complexity, and performs structural analysis on the representative document; S400 constructs a cache template for corresponding grouped document detection based on the information representing the document; S500 performs document data quality detection on non-representative documents in the group based on the cache template; When a new document is added to the group, S600 builds and triggers a cache update mechanism, updates the cache template, and completes data quality detection of the document.

2. The resource consumption intelligent monitoring method for document data quality detection according to claim 1 is characterized by: The specific steps of extracting metadata information and structural feature information of a document in S100 are as follows: S101 collects the documents for quality inspection and constructs the original document set D={d1,d2,d3,...,dn}, where d1,d2,d3,... and dn represent the first, second, third,... and nth documents respectively. For the xth document, x∈[1,n], the metadata information of the document is extracted by the document parser and recorded as Yx. The metadata extraction result is represented by the set Xy={Yx1,Yx2,Yx3,...,Yxm}, where Yx1,Yx2, Yx3,... and Yxm represent the first, second, third,... and mth metadata information extracted from the xth document respectively. The structural feature information Tx of the xth document is extracted by parsing the document content. The extraction result of the structural feature is expressed as a set Xt={Tx1,Tx2,Tx3,...,Txb}, where Tx1,Tx2,Tx3,... and Txb represent the first, second, third,... and bth structural feature information extracted from the xth document respectively; S102 combines the metadata information and structural feature information of the x-th document to form an information description of the x-th document, expressed as dx*={Xy,Xt}, and constructs a document set D^={d1*,d2*,d3*,...,dn*} represented by the document information, where d1*, d2*, d3*,... and dn* respectively represent the first, second, third,... and n-th documents containing the document information.

3. The resource consumption intelligent monitoring method for document data quality detection according to claim 2 is characterized by: The specific steps of grouping documents in S200 are: S201 normalizes the structural feature information and the numerical information in the metadata of the x-th document using a min-max method, converts the categorical information in the structural feature information and metadata into a one-hot encoding, obtains processed structural feature information Tx' and metadata information Yx', merges the processed structural feature information and metadata information of the x-th document, and constructs a final feature set dx'={Yx1', Yx2', Yx3', ..., Yxm', Tx1', Tx2', Tx3', ..., Txb'} of the x-th document; S202 calculates the similarity Sxz between the x-th document and the z-th document based on the data in the processed feature set, where z∈[1,n] and z≠x. The calculation formula is: ， Where k represents the data index of the metadata in the final feature set, and j represents the data index of the structural feature in the final feature set; S203 calculates the similarity between any two documents for all documents in the set D^ and performs statistical analysis on the similarity, calculates the mean μS and standard deviation σS of the similarity, and calculates the initial similarity threshold θmin based on the mean and standard deviation. The calculation formula is: ， The similarity between the xth and zth documents is compared with the threshold. When Sxz≥θmin, the xth document is taken as the core document of this group. The similarity between the remaining documents in the set D^ except the xth and zth documents and the xth document is compared with the threshold. Documents with a similarity greater than the threshold are divided into the group containing the xth document, and the remaining documents are not grouped for the time being. When Sxz<θmin, classification is not performed for the time being. For documents that are not classified, the threshold θmin is adjusted by introducing the adjustment factor λ to obtain a new threshold θmin1. The calculation formula of θmin1 is: θmin 1=μS+λσS, according to the threshold θmin1, the unclassified documents are compared with the threshold and classified again. The maximum number of adjustments is set to Tmax. When the maximum number of adjustments is reached, the adjustment is stopped. If the adjustment number reaches Tmax and is still not classified, the document is assigned to a separate group Gd+1. Finally, n documents are divided into several groups to obtain a group set G={G1,G2,G3,...,Gd,Gd+1}, where G1, G2, G3,...,Gd and Gd+1 represent the first, second, third,...,dth and separate groups respectively.

4. The resource consumption intelligent monitoring method for document data quality detection according to claim 3 is characterized by: The specific steps of performing structural analysis on the representative document in S300 are: S301 calculates the structural complexity Ci of the i-th document in the e-th group, e∈[1,d], i represents any document in the e-th group, and the calculation formula is: ， Where ω1, ω2, ω3, ... and ωb are the weights of each structural feature in the i-th document on the document complexity. The document with the largest structural complexity is selected from the e-th group as the representative document dc of the group. S302 analyzes the overall structure of the representative document in the e-th group through document element recognition, title hierarchy analysis, chapter division and chart positioning to obtain a structural analysis Tc, and then formulates corresponding title detection rules, paragraph format rules, table detection rules and chart detection rules for each structural element of the representative document to construct a detection rule set Rc, and then defines a title content template, paragraph content template, table content template and chart content template for each structural unit in the representative document, and defines title format constraints, paragraph format constraints, table format constraints and chart format constraints to obtain a content constraint template Pc.

5. The resource consumption intelligent monitoring method for document data quality detection according to claim 4 is characterized by: The specific steps of constructing a cache template for corresponding group document detection in S400 are: S401 divides the representative documents in the e-th group into several structural units Lj according to structural analysis, and constructs a structural unit set L={Lj1,Lj2,Lj3,...,Ljw}, where Lj1,Lj2,Lj3,... and Ljw represent the first, second, third,... and w-th structural units respectively. For the a-th structural unit in the set L, a∈[1,w], it contains the corresponding detection rule set Rca and content constraint template Pca; S402 stores the information of w structural units as a cache template Md, expressed as Md={(Lj1,Tc1,Rc1,Pc1),(Lj2,Tc2,Rc2,Pc2),(Lj3,Tc3,Rc3,Pc3),...,(Ljw,Tcw,Rcw,Pcw)}, where (Lj1,Tc1,Rc1,Pc1),(Lj2,Tc2,Rc2,Pc2),(Lj3,Tc3,Rc3,Pc3),... and (Ljw,Tcw,Rcw,Pcw) respectively represent the first, second, third,... and wth structural units and the cache template contents corresponding to the structural units, and the cache template Md is used as the cache template corresponding to the eth group.

6. The resource consumption intelligent monitoring method for document data quality detection according to claim 5 is characterized by: The specific steps of performing document data quality detection on the non-representative documents in the group in S500 are: S501: For other non-representative documents in the e-th group, the non-representative documents are divided into a number of structural units based on structural analysis. For each structural unit in the non-representative document, if there is a structural unit in the non-representative document that corresponds to the structural unit in the cache template, the detection rule set and content constraint template of the corresponding structural unit in the cache template are directly reused; S502 When the structural unit in the non-representative document does not exist in the cache template, the corresponding structural unit is marked as a new structural unit, and a temporary detection rule set and content constraint template are generated as a temporary cache, added to the cache template, and a temporary cache survival time threshold ξ is set. When the temporary cache detection is not triggered within the threshold time, the temporary cache is deleted.

7. The resource consumption intelligent monitoring method for document data quality detection according to claim 6 is characterized by: The specific steps for building and triggering the cache update mechanism in S600 are: S601: For the newly added document requiring data quality inspection, calculate the similarity with the representative documents in each group, select the group to which the representative document with the maximum similarity belongs, and add the newly added document to the group; S602 For the e-th group, when a new document is added, the threshold value of the number of new documents in the group is set to Y. When the number of new documents added to the e-th group exceeds the threshold, the structural complexity of the Y new documents in the e-th group is calculated and compared with the structural complexity of the representative document dc in the group. When the structural complexity of the new document is less than or equal to the structural complexity of the representative document dc, the cache template constructed by the representative document is used to perform data quality detection on the new document. When the structural complexity of the new document is greater than the structural complexity of the representative document dc, the new document is used as the new representative document dc*, the new representative document is divided into structural units, and a new cache template Md* is constructed as the cache template of the e-th group. The cache template Md is replaced and the data quality detection is performed on the new document.

8. A resource consumption intelligent monitoring system for document data quality detection, applied to a resource consumption intelligent monitoring method for document data quality detection according to any one of claims 1 to 7, characterized in that: The system includes a feature extraction module, a document grouping module, a parsing module, a cache template building module, a document detection module and a cache update module; The feature extraction module is used to extract metadata information and structural feature information of the document for data quality detection; The document grouping module is used to normalize the metadata information and structural feature information of the documents and group the documents; The parsing module is used to calculate the structural complexity of the documents in the group, select representative documents according to the structural complexity, and perform structural analysis on the representative documents; The cache template construction module is used to construct a cache template for corresponding group document detection based on the information representing the document; The document detection module is used to perform document data quality detection on non-representative documents in the group based on the cache template; The cache update module is used to build and trigger a cache update mechanism when a new document is added to a group.

9. The resource consumption intelligent monitoring system for document data quality detection according to claim 8, characterized in that: The parsing module includes a complexity calculation unit and a representative document identification unit; The complexity calculation unit is used to calculate the structural complexity of the documents in the group; The representative document identification unit is used to identify the document with the highest structural complexity in the group as the representative document.

Images