Intelligent monitoring system and method for resource consumption for document data quality detection

Abstract

The application discloses a resource consumption intelligent monitoring system and method for document data quality detection, relates to the technical field of document detection, extracts metadata information and structural feature information of a document for data quality detection, and constructs an information document set; carries out normalization processing on the metadata information and the structural feature information of the document, and groups the document; calculates the structural complexity of the documents in a group based on the grouping, selects a representative document according to the structural complexity, and carries out structural analysis on the representative document; constructs a cache template for detecting the documents in the group according to the information of the representative document; carries out document data quality detection on non-representative documents in the group based on the cache template; when a new document is added to the group, a cache updating mechanism is constructed and triggered, the cache template is updated, and the data quality detection on the document is completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of document detection, and specifically to a resource consumption intelligent supervision system and method for document data quality detection. BACKGROUND

[0002] In modern document processing and quality detection, with the increase in the number of document types and the improvement in content complexity, traditional document quality detection methods usually rely on checking each part of the document one by one, including metadata verification, text content consistency check, format check and structure analysis, etc. Although these methods can ensure document quality, when the amount of documents is large or the content is complex, the processing process will consume a large amount of computing resources and time, especially when multiple documents need to be detected for multiple rounds, the efficiency and performance bottleneck are particularly obvious.

[0003] Current document detection methods often face the problems of high resource consumption and low processing efficiency, especially the traditional detection method relies on re-parsing and analyzing the entire document each time. These methods usually use rule engines and fixed algorithms for full-text processing. Whenever the document changes, the document needs to be analyzed and detected completely, resulting in waste of computing resources. In addition, in the traditional method, resource management is mostly based on fixed rule allocation, which cannot dynamically adjust resources according to the complexity of the document or the processing stage, so it cannot effectively avoid performance problems such as high load running and long waiting time.

[0004] Therefore, there is an urgent need for a resource consumption intelligent supervision system and method for document data quality detection to solve the above problems. SUMMARY

[0005] The purpose of the present application is to provide a method that solves the problems in the prior art.

[0006] To achieve the above purpose, the present application provides the following technical scheme: a resource consumption intelligent supervision method for document data quality detection, the method comprising the following steps:

[0007] S100 extracting metadata information and structure feature information of the document to be detected for data quality, and constructing an information document set;

[0008] Further, the specific steps of extracting the metadata information and structure feature information of the document are as follows:

[0009] S101 collects the documents for quality detection, and constructs an original document set D={d1, d2, d3,...,dn}, wherein d1, d2, d3,... and dn represent the 1st, 2nd, 3rd,... and nth document respectively. For the xth document, x∈[1,n], the metadata information of the document is extracted by a document parser and denoted as Yx. The metadata is the basic information of the document, including but not limited to the following contents: document type, document size, document page number, creation time / modification time and author. The extraction result of the metadata is denoted as a set Xy={Yx1,Yx2,Yx3,...,Yxm}, wherein Yx1, Yx2, Yx3,... and Yxm represent the 1st, 2nd, 3rd,... and mth metadata information of the xth document respectively. The structural feature information Tx of the xth document is extracted by parsing the document content. The document structural feature refers to the layout, relationship and semantic information of each element in the document. The extraction method includes but is not limited to natural language processing (NLP), regular expression, layout analysis, optical character recognition and machine learning and deep learning. The extraction result of the structural feature is denoted as a set Xt={Tx1,Tx2,Tx3,...,Txb}, wherein Tx1, Tx2, Tx3,... and Txb represent the 1st, 2nd, 3rd,... and bth structural feature information of the xth document respectively.

[0010] S102 combines the metadata information and the structural feature information of the xth document to form the information description of the xth document, denoted as dx*={Xy,Xt}. A document set D^ represented by the document information is constructed, wherein d1*, d2*, d3*,... and dn* represent the 1st, 2nd, 3rd,... and nth document containing document information respectively.

[0011] S200 normalizes the metadata information and the structural feature information of the document, and groups the documents.

[0012] Further, the specific steps of grouping the documents are as follows:

[0013] S201 For the numerical information in the structural feature information and the metadata information of the xth document, the numerical information in the structural feature includes the number of sections, the number of charts, the title level depth, the number of table rows and columns, etc., and the numerical information in the metadata includes the document size, the number of document pages, and the creation and modification timestamps, etc. The minimum-maximum method is used for normalization processing. For the categorical information in the structural feature information and the metadata information, the categorical information in the structural feature includes the icon type, the title style, and the paragraph alignment, etc., and the categorical information in the metadata includes the document type, the author's department, and the document security level, etc. The categorical information is converted into one-hot encoding to obtain the processed structural feature information Tx’ and the metadata information Yx’. The processed structural feature information and the metadata information of the xth document are merged to construct the final feature set dx’ of the xth document dx’ = {Yx1’, Yx2’, Yx3’,..., Yxm’, Tx1’, Tx2’, Tx3’,..., Txb’};

[0014] S202 Based on the data in the processed feature set, the similarity Sxz between the xth document and the zth document is calculated, z ∈ [1, n] and z ≠ x, and the calculation formula is:

[0015] ,

[0016] where k represents the data index of the metadata in the final feature set, and j represents the data index of the structural feature in the final feature set;

[0017] S203 For all documents in the set D^, the similarity values between any two documents are calculated and statistical analysis is performed on the similarity values. The mean μS and the standard deviation σS of the similarity values are calculated. Based on the mean and the standard deviation, the initial similarity threshold θmin is calculated, and the calculation formula is:

[0018] ,

[0019] The similarity between the xth and zth documents is compared with a threshold value, when Sxz≥θmin, the xth document is taken as the core document of the group, the similarity between the remaining documents of the set D^ excluding the xth and zth documents and the xth document is compared with the threshold value, the documents greater than the threshold value are divided into the group containing the xth document, the remaining documents are temporarily not grouped, when Sxz<θmin, the classification is also temporarily not performed, for the documents not classified, by introducing an adjustment factor λ, the value of λ is dynamically adjusted according to the real-time classification performance, such as accuracy, recall rate and the like, and the reduction ratio of the unclassified documents, when the classification effect is good and the unclassified documents are significantly reduced, λ can be appropriately increased, and the threshold value is increased accordingly, so as to meet the requirements of avoiding false classification due to loose classification standard and ensuring the classification quality in the case of significant reduction of unclassified documents; when the classification effect is poor or the unclassified documents are reduced slowly, λ is reduced, and the corresponding threshold value is reduced accordingly, at this time, more documents can meet the threshold value requirement, thereby reducing the number of unclassified documents and improving the classification efficiency, the threshold value θmin is adjusted to obtain a new threshold value θmin1, the calculation formula of θmin1 is: θmin1=μS+λσS, according to the threshold value θmin1, the documents not classified are classified again by comparing with the threshold value, the maximum adjustment number is set as Tmax, when the maximum adjustment number is reached, the adjustment is stopped, if the adjustment number reaches Tmax and the documents are still not classified, the documents are divided into a separate group Gd+1, finally, the n documents are divided into a plurality of groups to obtain a group set G={G1,G2,G3,...,Gd,Gd+1}, wherein G1,G2,G3,...,Gd and Gd+1 respectively represent the 1st, 2nd, 3rd,..., dth and separate groups;

[0020] S300 calculates the structural complexity of the documents in each group based on the grouping, selects a representative document according to the structural complexity, and performs structural analysis on the representative document;

[0021] Further, the specific steps of performing structural analysis on the representative document are as follows:

[0022] S301 calculates the structural complexity Ci of the ith document in the e th group, e∈[1,d], i represents any one document in the e th group, and the calculation formula is:

[0023] ,

[0024] Wherein ω1, ω2, ω3, … and ωb are the weights of each structural feature in the i-th document on the impact of document complexity, the weight selection should be based on the actual impact of document structural features and specific needs, can use expert knowledge, statistical methods, machine learning models and other means to allocate appropriate weights for each feature, now a method is described: by collecting a number of historical document detection data, record the corresponding calculation resource consumption of each structural feature, such as parsing time and memory usage, calculate the consumption resource proportion of a single structural feature, the calculation formula is: Wherein Represents the consumption resource proportion of any single structural feature, Tre is the average resource consumption of any single structural feature, Represents the total resource consumption of all structural features, the weight is determined according to the resource consumption proportion; Select the document with the maximum structural complexity from the e-th group as the representative document dc of the group;

[0025] S302, for the representative document in the e-th group, the overall structure of the representative document is parsed through document element recognition, title level analysis, chapter division and chart positioning, and the structure parsing Tc is obtained, the document element recognition (cover, table of contents, chapter, paragraph, chart, table); Title level analysis (based on font style, alignment); Chapter division (according to "chapter X" mark); Chart positioning (record position and type), then make corresponding title detection rules, paragraph format rules, table detection rules and chart detection rules for each structural element of the representative document, construct detection rule set Rc, define a set of specific detection rules for each document structure unit (such as title, paragraph, table, chart), these rules include how to identify title, paragraph format, table structure, and chart format, etc., then define title content template, paragraph content template, table content template and chart content template for each structural unit in the representative document, and define title format constraint, paragraph format constraint, table format constraint and chart format constraint, get content constraint template Pc;

[0026] S400, according to the information of the representative document, construct the cache template for corresponding grouped document detection;

[0027] Further, the specific steps of constructing the cache template for corresponding grouped document detection are:

[0028] S401 divides the representative document in the e-th group into several structural units Lj according to structural parsing, the document is divided into multiple structural units, and the units are parts of the document, such as: title: first-level title, second-level title, etc., paragraph: each text paragraph, table: independent part of each table, chart: individual part of each chart, the structural units are identified based on the structural parsing result of the document through title level, chapter division and the like, and a structural unit set L = {Lj1, Lj2, Lj3,..., Ljw} is constructed, wherein Lj1, Lj2, Lj3,..., and Ljw represent the 1st, 2nd, 3rd,..., and w-th structural unit, respectively, and for the a-th structural unit in the set L, a ∈ [1, w], the corresponding detection rule set Rca and the content constraint template Pca are included;

[0029] S402 stores the information of the w structural units as a cache template Md, which is represented as Md = {(Lj1, Tc1, Rc1, Pc1), (Lj2, Tc2, Rc2, Pc2), (Lj3, Tc3, Rc3, Pc3),..., (Ljw, Tcw, Rcw, Pcw)}, wherein (Lj1, Tc1, Rc1, Pc1), (Lj2, Tc2, Rc2, Pc2), (Lj3, Tc3, Rc3, Pc3),..., and (Ljw, Tcw, Rcw, Pcw) represent the 1st, 2nd, 3rd,..., and w-th structural unit and the cache template content corresponding to the structural unit, and the cache template Md is used as the cache template corresponding to the e-th group;

[0030] The information of the structural units, such as the parsing result, the detection rule, and the constraint template, is stored through the cache template, which avoids re-parsing and rule construction with large resource consumption each time, can effectively reduce the computing load and memory consumption of the system in the document detection process, and can flexibly use the information in the cache template according to the complexity and type of the document, so that unnecessary resource consumption is reduced;

[0031] S500 performs document data quality detection on the non-representative documents in the group based on the cache template;

[0032] Further, the specific steps of performing document data quality detection on the non-representative documents in the group are as follows:

[0033] S501 for other non-representative documents in the e-th group, divides the non-representative document into several structural units according to structural parsing, for each structural unit in the non-representative document, when there is a corresponding structural unit in the cache template, directly reuses the detection rule set and the content constraint template of the corresponding structural unit in the cache template;

[0034] S502 When the structural unit in the non-representative document does not exist in the cache template, mark the corresponding structural unit as a new structural unit, generate a temporary detection rule set and a content constraint template, as a temporary cache, add to the cache template, set the temporary cache survival time threshold ξ, and delete the temporary cache when no temporary cache detection is triggered within the threshold time; It needs to be further explained that a series of constraints are introduced in the generation of the temporary detection rule set and the survival time threshold ξ. First, limit the frequency of rule set generation to avoid excessive frequent creation of new caches, and evaluate the survival time according to the effectiveness of the cache. If the temporary cache does not trigger detection, it is automatically deleted when it exceeds the set threshold. Secondly, avoid overly complex or repetitive rule sets by merging similar rule sets and limiting the number of rule set contents to ensure efficient use of system resources. In addition, a dynamic adjustment of the cache number, frequency and effectiveness checking mechanism is adopted to ensure the stability of the temporary cache system and effectively prevent resource waste;

[0035] Through dynamic cache management and adaptive rule generation mechanism, the document processing is more flexible and efficient, not only improves the efficiency of document processing and reduces the consumption of computing and storage resources, but also enhances the adaptability of the system to new document structure and format. Through intelligent management of the survival time and update rules of the cache, the system can continuously optimize and improve the accuracy and efficiency of detection, ensuring high-quality document processing and management;

[0036] S600 When a new document is added to the group, build and trigger the cache update mechanism to update the cache template and complete the data quality detection of the document;

[0037] Further, the specific steps of building and triggering the cache update mechanism are:

[0038] S601 For the newly added document that needs data quality detection, calculate the similarity with each representative document in the group, select the group corresponding to the representative document with the maximum similarity, and add the new document to the group;

[0039] S602 For the e-th group, when a new document is added, set the threshold of the number of newly added documents in the group as Y, the value of Y can be adjusted according to the system load, when the system load is high, Y can be appropriately increased, and the frequent organization structure adjustment can be reduced to avoid excessive consumption of resources; and when the system load is low, Y can be set to a smaller value, when the number of new documents added in the e-th group exceeds the threshold, the structural complexity of Y newly added documents in the e-th group is calculated, and the structural complexity of the representative document dc in the group is compared, when the structural complexity of the newly added document is less than or equal to the structural complexity of the representative document dc, the cache template constructed by the representative document is used for data quality detection of the newly added document, when the structural complexity of the newly added document is greater than the structural complexity of the representative document dc, the newly added document is taken as a new representative document dc*, the new representative document is divided into structural units, and a new cache template Md* is constructed as the cache template of the e-th group, the cache template Md is replaced, and the data quality detection of the newly added document is performed;

[0040] When the data quality detection is performed through the cache template, cache invalidation may also exist, which is caused by many reasons, including document content or structure change, format update, detection rule modification, system environment change, cache space limitation, time expiration, similarity change, user operation change, and multi-version or multi-source document processing. In order to solve these problems, a dynamic cache management mechanism can be used, combined with document version and structure analysis, to update the cache content in real time; the cache expiration time is set, and the invalid cache is cleaned up regularly; the detection rule change and the document update are intelligently monitored;

[0041] By dynamically selecting whether to update the cache template according to the structural complexity of the newly added document, the efficient document processing is ensured, and the flexible adaptation ability of the system to the change of the structural complexity of the document is ensured, the use of the computing and storage resources is effectively optimized, the accuracy and consistency of the document quality detection are improved, and the efficiency of the document processing is maintained;

[0042] In order to better realize the above method, a resource consumption intelligent monitoring system for document data quality detection is also proposed, the system includes a feature extraction module, a document grouping module, an analysis module, a cache template construction module, a document detection module and a cache update module;

[0043] The feature extraction module is used for extracting the metadata information and the structural feature information of the document for data quality detection;

[0044] The document grouping module is used for normalizing the metadata information and the structural feature information of the document, and grouping the document;

[0045] The parsing module is used for calculating the structural complexity of the grouped documents, selecting a representative document according to the structural complexity, and performing structural parsing on the representative document;

[0046] The parsing module comprises a complexity calculation unit and a representative document identification unit;

[0047] The complexity calculation unit is used for calculating the structural complexity of the grouped documents;

[0048] The representative document identification unit is used for identifying the document with the highest structural complexity in the group as the representative document;

[0049] The cache template construction module is used for constructing a cache template for detecting the corresponding grouped documents according to the information of the representative document;

[0050] The document detection module is used for performing document data quality detection on the non-representative documents in the group based on the cache template;

[0051] The cache update module is used for constructing and triggering a cache update mechanism when a new document is added to the group.

[0052] Compared with the prior art, the present application has the following advantages:

[0053] 1. By analyzing the structural features of the documents and storing the cache templates, the existing templates can be reused in the detection process, thereby avoiding the high overhead of processing resources caused by reanalyzing the document structure each time, and greatly improving the processing efficiency. When the complexity of the new documents changes, the system can automatically select appropriate cache templates for processing according to the structural complexity of the documents, which enables the system to efficiently process a large number of documents with different complexities and improves the overall processing speed.

[0054] 2. By dynamically adjusting the used cache templates according to the structural complexity of the documents, the system can flexibly cope with documents with different structures to ensure adaptability to different document structures. For each document, especially the new documents, the system can automatically generate or adjust the cache templates according to their structural complexity, thereby avoiding the insufficient adaptability caused by the application of static templates and ensuring the flexibility of document detection.

[0055] 3. As the number of documents increases and the types of documents diversify, the system can automatically adapt to the complexity and structure of the new documents. By dynamically updating the cache templates, the system can ensure effective processing of the growing and changing document library. Through analysis of the new documents, the document processing flow can be gradually optimized. As more documents are added, the system can continuously improve efficiency and accuracy, and has the ability of self-optimization. BRIEF DESCRIPTION OF DRAWINGS

[0056] Fig. 1 It is a structural schematic diagram of the resource consumption intelligent monitoring system for document data quality detection.

[0057] Fig. 2 The flowchart of the resource consumption intelligent supervision method for document data quality detection of the present application;

[0058] Fig. 3 The embodiment diagram of the resource consumption intelligent supervision method for document data quality detection of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0060] Embodiment: As shown in the present application, a technical solution is provided, which is a resource consumption intelligent supervision method for document data quality detection. The method comprises the following steps: Figs. 1-3

[0061] S100: Extracting metadata information and structural feature information of the document for data quality detection, and constructing an information document set.

[0062] Further, the specific steps of extracting the metadata information and the structural feature information of the document are as follows:

[0063] ​S101 collects the documents for quality detection, and constructs an original document set D={d1, d2, d3,...,dn}, wherein d1, d2, d3,... and dn represent the 1st, 2nd, 3rd,... and nth document respectively. For the xth document, x∈[1,n], the metadata information of the document is extracted by a document parser and denoted as Yx. The metadata is the basic information of the document, including but not limited to the following contents: document type, document size, document page number, creation time / modification time and author. The extraction result of the metadata is denoted as a set Xy={Yx1,Yx2,Yx3,...,Yxm}, wherein Yx1, Yx2, Yx3,... and Yxm represent the 1st, 2nd, 3rd,... and mth metadata information of the xth document respectively. The structural feature information Tx of the xth document is extracted by parsing the document content. The document structural feature refers to the layout, relationship and semantic information of each element in the document. The extraction method includes but is not limited to natural language processing (NLP), regular expression, layout analysis, optical character recognition and machine learning and deep learning. The extraction result of the structural feature is denoted as a set Xt={Tx1,Tx2,Tx3,...,Txb}, wherein Tx1, Tx2, Tx3,... and Txb represent the 1st, 2nd, 3rd,... and bth structural feature information of the xth document respectively.

[0064] S102 combines the metadata information and the structural feature information of the xth document to form the information description of the xth document, denoted as dx*={Xy,Xt}. A document set D^ represented by the document information is constructed, wherein d1*, d2*, d3*,... and dn* represent the 1st, 2nd, 3rd,... and nth document containing document information respectively.

[0065] S200 normalizes the metadata information and the structural feature information of the document, and groups the documents.

[0066] Further, the specific steps of grouping the documents are as follows:

[0067] S201 For the numerical information in the structural feature information and the metadata information of the xth document, the numerical information in the structural feature includes the number of sections, the number of charts, the title level depth, the number of table rows and columns, etc., and the numerical information in the metadata includes the document size, the number of document pages, and the creation and modification timestamps, etc. The minimum-maximum method is used for normalization processing. For the categorical information in the structural feature information and the metadata information, the categorical information in the structural feature includes the icon type, the title style, and the paragraph alignment, etc., and the categorical information in the metadata includes the document type, the author's department, and the document security level, etc. The categorical information is converted into one-hot encoding to obtain the processed structural feature information Tx’ and the metadata information Yx’. The processed structural feature information and the metadata information of the xth document are merged to construct the final feature set dx’ of the xth document dx’ = {Yx1’, Yx2’, Yx3’,..., Yxm’, Tx1’, Tx2’, Tx3’,..., Txb’};

[0068] S202 Based on the data in the processed feature set, the similarity Sxz between the xth document and the zth document is calculated, z ∈ [1, n] and z ≠ x, and the calculation formula is:

[0069] ,

[0070] where k represents the data index of the metadata in the final feature set, and j represents the data index of the structural feature in the final feature set;

[0071] S203 For all documents in the set D^, the similarity values between any two documents are calculated and statistical analysis is performed on the similarity values. The mean μS and the standard deviation σS of the similarity values are calculated. Based on the mean and the standard deviation, the initial similarity threshold θmin is calculated, and the calculation formula is:

[0072] ,

[0073] The similarity between the xth and zth documents is compared with the threshold value, when Sxz≥0min, the xth document is taken as the core document of this group, the similarity between the remaining documents of the set D^ excluding the xth and zth documents and the xth document is compared with the threshold value, the documents greater than the threshold value are divided into the group containing the xth document, the remaining documents are temporarily not grouped, when Sxz<0min, the same is temporarily not classified, for the documents not classified, by introducing an adjustment factor λ, λ∈[0, 1], by monitoring the classification performance indicators accuracy P, recall rate R and the reduction ratio of unclassified documents ΔU (ΔU=(unclassified number before adjustment-unclassified number after adjustment) / unclassified number before adjustment, the accuracy P indicates the proportion of “the number of documents correctly divided into a certain group” to “the actual total number of documents in the group” after grouping, which is used to measure the accuracy of grouping (to avoid irrelevant documents being divided into the group), the recall rate R indicates the proportion of “the number of documents correctly divided into a certain group” to “the total number of documents that should be divided into the group” after grouping, which is used to measure the integrity of grouping (to avoid missing documents that should be divided into the group)), set the judgment threshold, such as P≥85%, R≥80% is good classification effect, ΔU≥30% is significant reduction, when “P≥85% and R≥80%” and “ΔU≥30%” are met at the same time, it means that the effect is good and the unclassified documents are significantly reduced under the current classification standard, λ increases by 0.1 (upper limit 1), based on the significant reduction of unclassified documents, the strict classification standard is maintained to ensure the classification quality; when “P<70% or R<65%” (poor effect) or “ΔU<10%” (slow reduction), it indicates that there are problems under the current classification standard, such as too high threshold value leading to too many misclassifications or unclassified documents being difficult to reduce, λ decreases by 0.2 (lower limit 0), the similarity of more documents can meet the threshold requirement, so as to relax the classification standard to reduce misclassification and promote the division of unclassified documents; in other intermediate states (P is between 70% and 85%, the recall rate R is between 65% and 80%, and the unclassified document reduction ratio ΔU is between 10% and 30%), λ is fine tuned by ±0.05, the classification quality and the unclassified reduction speed are balanced, λ is dynamically adapted according to the real-time classification effect, until the maximum adjustment times Tmax is reached; adjust the threshold value 0min to obtain a new threshold value 0min1, the calculation formula of 0min1 is: 0min1=μS+λσS, according to the threshold value 0min1, the unclassified documents are classified again by comparing with the threshold value, the maximum adjustment times is set as Tmax, when the maximum adjustment times is reached, the adjustment is stopped, if the adjustment times reaches Tmax and the documents are still not classified, the documents are divided into a separate group Gd+1, finally n documents are divided into a plurality of groups to obtain a group set G={G1, G2, G3,..., Gd, Gd+1}, wherein G1, G2, G3,..., Gd and Gd+1 respectively represent the 1st, 2nd, 3rd,..., dth and separate groups;

[0074] S300 calculates the structural complexity of the documents in each group based on the grouping, selects a representative document from each group according to the structural complexity, and performs structural analysis on the representative document;

[0075] Further, the specific steps of performing structural analysis on the representative document are as follows:

[0076] S301 calculates the structural complexity Ci of the i-th document in the e-th group, where e [1, d], i represents any one document in the e-th group, and the calculation formula is:

[0077]

[0078] where ω1, ω2, ω3, …, and ωb are the weights of each structural feature in the i-th document on the document complexity, and the weights should be selected according to the actual influence and specific requirements of the document structure features. Appropriate weights can be assigned to each feature using various means such as expert knowledge, statistical methods, machine learning models, etc. One method is described as follows: By collecting a number of historical document detection data, record the corresponding calculation resource consumption of each structural feature, such as parsing time and memory occupation, calculate the resource consumption proportion of a single structural feature, and the calculation formula is: where represents the resource consumption proportion of any single structural feature, Tre is the average resource consumption of any single structural feature, represents the total resource consumption of all structural features, and the weights are determined according to the resource consumption proportion;

[0079] Suppose the structural features of the i-th document include: Ti1: number of title levels, Ti2: number of section numbers, Ti3: number of tables and charts, and Ti4: number of pages. In the historical detection process, the resource consumption is respectively: parsing time (Ti1: 15, Ti2: 15, Ti3: 10, Ti4: 5) in seconds, and memory occupation (Ti1: 50, Ti2: 70, Ti3: 30, Ti4: 20) in MB. The data is standardized using the maximum-minimum standardization method, and then according to the above method, the resource consumption proportion of each structural unit is obtained. The weight of each structural feature can be determined as its proportion value by calculating the resource consumption proportion. Then, the document with the maximum structural complexity is selected from the e-th group as the representative document dc of the group;

[0080] ​S302, for the representative document in the e-th group, the overall structure of the representative document is parsed by document element recognition, title level analysis, chapter division and chart positioning to obtain structure parsing Tc, document element recognition (cover, table of contents, chapter, paragraph, chart, table); title level analysis (based on font style, alignment); chapter division (according to "chapter X" mark); chart positioning (record location and type), then formulate corresponding title detection rules, paragraph format rules, table detection rules and chart detection rules for each structural element of the representative document, construct detection rule set Rc, define a set of specific detection rules for each document structure unit (such as title, paragraph, table, chart), which includes how to identify title, paragraph format, table structure, and chart format, etc., then define title content template, paragraph content template, table content template and chart content template for each structural unit in the representative document, and define title format constraint, paragraph format constraint, table format constraint and chart format constraint, to obtain content constraint template Pc; define content template and format constraint for each structural unit (such as title, paragraph, table, chart, etc.), for example, for the title, define which content should be regarded as a legal title, and constrain its font, size, alignment, etc.; for the paragraph, define the layout and format of the paragraph, title content template: such as "chapter X: chapter title", paragraph content template: such as the basic sentence structure that the paragraph should contain, format requirements, table content template: columns and rows that should be contained in the table, font, alignment, etc. constraints, chart content template: chart type, label, and how to display data;

[0081] S400, according to the information of the representative document, a cache template for corresponding grouped document detection is constructed;

[0082] Further, the specific steps of constructing a cache template for corresponding grouped document detection are:

[0083] S401, the representative document in the e-th group is divided into several structural units Lj according to the structure parsing, the document is divided into multiple structural units, these units are parts of the document, such as: title: first-level title, second-level title, etc., paragraph: each text paragraph, table: each table independent part, chart: each chart individual part, these structural units are identified based on the structure parsing results of the document, through title level, chapter division, etc., to construct a structural unit set L = {Lj1, Lj2, Lj3,..., Ljw}, wherein Lj1, Lj2, Lj3,..., and Ljw represent the first, second, third,..., and w-th structural units, for the a-th structural unit in which a ∈ [1, w], contains corresponding detection rule set Rca and content constraint template Pca;

[0084] S402 stores the information of the w structural units as a cache template Md, denoted as Md={(Lj1, Tc1, Rc1, Pc1), (Lj2, Tc2, Rc2, Pc2), (Lj3, Tc3, Rc3, Pc3),..., (Ljw, Tcw, Rcw, Pcw)}, wherein (Lj1, Tc1, Rc1, Pc1), (Lj2, Tc2, Rc2, Pc2), (Lj3, Tc3, Rc3, Pc3),... and (Ljw, Tcw, Rcw, Pcw) respectively represent the 1st, 2nd, 3rd,... and wth structural unit and the cache template content corresponding to the structural unit, and the cache template Md is taken as the cache template corresponding to the e-th group;

[0085] The information of the structural units, i.e. the analysis result, the detection rule and the constraint template, is stored through the cache template, so that the analysis and rule construction with large resource consumption are not performed every time of detection, the calculation load and the memory consumption of the system in the document detection process can be effectively reduced, the information in the cache template can be flexibly used according to the complexity and type of the document, and comprehensive calculation is not performed every time, so that unnecessary resource consumption is reduced;

[0086] S500 performs document data quality detection on the non-representative documents in the group based on the cache template;

[0087] Further, the specific steps of performing document data quality detection on the non-representative documents in the group are as follows:

[0088] S501 for other non-representative documents in the e-th group, divides the non-representative documents into several structural units according to structural analysis, and for each structural unit in the non-representative document, when there is a corresponding structural unit in the cache template, directly reuses the detection rule set and the content constraint template of the corresponding structural unit in the cache template;

[0089] S502 When the structural unit in the non-representative document does not exist in the cache template, mark the corresponding structural unit as a new structural unit, generate a temporary detection rule set and a content constraint template, as a temporary cache, add to the cache template, set the temporary cache survival time threshold ξ, when there is no temporary cache detection triggered within the threshold time, delete the temporary cache; It needs to be further explained that: a series of constraints are introduced in the generation of the temporary detection rule set and the survival time threshold ξ, first, limit the frequency of rule set generation to avoid excessive frequent creation of new caches, and evaluate the survival time according to the effectiveness of the cache, if the temporary cache does not trigger detection, automatically delete when it exceeds the set threshold, second, avoid too complex or repeated rule sets, by merging similar rule sets and limiting the number of rule set contents, ensure efficient use of system resources, in addition, adopt dynamic adjustment of cache number, frequency and effectiveness checking mechanism, ensure the stability of the temporary cache system and effectively prevent resource waste;

[0090] Through the dynamic cache management and adaptive rule generation mechanism, the document processing is more flexible and efficient, not only improves the efficiency of document processing, reduces the consumption of computing and storage resources, but also enhances the adaptability of the system to new document structure and format, through intelligent management of the survival time and update rules of the cache, the system can continuously optimize and improve the accuracy and efficiency of detection, ensure high-quality document processing and management;

[0091] S600 When a new document is added to the group, build and trigger the cache update mechanism to update the cache template and complete the data quality detection of the document;

[0092] Further, the specific steps of building and triggering the cache update mechanism are:

[0093] S601 For the newly added document that needs data quality detection, calculate the similarity with each representative document in the group, select the group corresponding to the representative document with the maximum similarity, and add the new document to the group;

[0094] S602 For the e-th group, when a new document is added, set the threshold of the number of newly added documents in the group as Y, the value of Y can be adjusted according to the system load, when the system load is high, the value of Y can be appropriately increased, for example, set to 15% of the number of documents in the group, reduce the frequent group structure adjustment, to avoid excessive consumption of resources; while the system load is low, Y can be set to a smaller value, such as 5% of the number of documents in the group, when the number of new documents added in the e-th group exceeds the threshold, calculate the structural complexity of Y newly added documents in the e-th group, and compare it with the structural complexity of the representative document dc in the group, when the structural complexity of the newly added document is less than or equal to the structural complexity of the representative document dc, the cache template constructed by the representative document is used to detect the data quality of the newly added document, when the structural complexity of the newly added document is greater than the structural complexity of the representative document dc, the newly added document is taken as a new representative document dc*, the new representative document is divided into structural units, and a new cache template Md* is constructed as the cache template of the e-th group, the cache template Md is replaced, and the data quality of the newly added document is detected;

[0095] When data quality detection is performed through the cache template, cache invalidation may also exist, which is caused by many reasons, including document content or structure change, format update, detection rule modification, system environment change, cache space limitation, time expiration, similarity change, user operation change, and multi-version or multi-source document processing, etc. To solve these problems, a dynamic cache management mechanism can be used, combined with document version and structure analysis, to update the cache content in real time; set the cache expiration time, and clean up the invalid cache regularly; intelligently monitor the changes of detection rules and document updates;

[0096] By dynamically selecting whether to update the cache template according to the structural complexity of the newly added document, both efficient document processing and flexible adaptation of the system to changes in document structural complexity are ensured, which can effectively optimize the use of computing and storage resources, improve the accuracy and consistency of document quality detection, and maintain high efficiency of document processing;

[0097] To better implement the above method, a resource consumption intelligent monitoring system for document data quality detection is also proposed, which includes a feature extraction module, a document grouping module, an analysis module, a cache template construction module, a document detection module, and a cache update module.

[0098] The feature extraction module is used to extract the metadata information and structural feature information of the document for data quality detection;

[0099] The document grouping module is used to normalize the metadata information and structural feature information of the document, and group the document;

[0100] The parsing module is used for calculating the structural complexity of the grouped documents, selecting a representative document according to the structural complexity, and performing structural parsing on the representative document;

[0101] The parsing module comprises a complexity calculating unit and a representative document identifying unit;

[0102] The complexity calculating unit is used for calculating the structural complexity of the grouped documents;

[0103] The representative document identifying unit is used for identifying the document with the highest structural complexity in the group as the representative document;

[0104] The cache template constructing module is used for constructing a cache template for detecting the corresponding grouped documents according to the information of the representative document;

[0105] The document detecting module is used for performing document data quality detection on the non-representative documents in the group based on the cache template;

[0106] The cache updating module is used for constructing and triggering a cache updating mechanism when a new document is added to the group.

[0107] It is apparent for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and range of the equivalent elements of the claims are intended to be embraced in the present application. Any reference signs in the claims should not be considered as limiting the involved claims.

Claims

1. An intelligent resource consumption monitoring method for document data quality detection, characterized by: S100 extracts metadata and structural feature information from documents undergoing data quality inspection and constructs an information document collection; S200 normalizes the metadata information and structural feature information of the documents and groups the documents. The specific steps of grouping the documents are as follows: S201 normalizes the structural feature information and the numerical information in the metadata of the x-th document using a min-max method, converts the categorical information in the structural feature information and metadata into a one-hot encoding, obtains processed structural feature information Tx' and metadata information Yx', merges the processed structural feature information and metadata information of the x-th document, and constructs a final feature set dx'={Yx1', Yx2', Yx3', ..., Yxm', Tx1', Tx2', Tx3', ..., Txb'} of the x-th document; S202 calculates the similarity Sxz between the x-th document and the z-th document based on the data in the processed feature set, where z∈[1,n] and z≠x. The calculation formula is: ， Where k represents the data index of the metadata in the final feature set, and j represents the data index of the structural feature in the final feature set; S203 calculates the similarity between any two documents for all documents in the set D^ and performs statistical analysis on the similarity, calculates the mean μS and standard deviation σS of the similarity, and calculates the initial similarity threshold θmin based on the mean and standard deviation. The calculation formula is: ， The similarity between the xth and zth documents is compared with the threshold. When Sxz≥θmin, the xth document is taken as the core document of this group. The similarity between the remaining documents in the set D^ except the xth and zth documents and the xth document is compared with the threshold. Documents with a similarity greater than the threshold are divided into the group containing the xth document, and the remaining documents are not grouped for the time being. When Sxz<θmin, classification is not performed for the time being. For documents that are not classified, the threshold θmin is adjusted by introducing the adjustment factor λ to obtain a new threshold θmin1. The calculation formula of θmin1 is: θmin 1=μS+λσS, according to the threshold θmin1, the unclassified documents are compared with the threshold and classified again. The maximum number of adjustments is set to Tmax. When the maximum number of adjustments is reached, the adjustment is stopped. If the document is still not classified after the adjustment number reaches Tmax, it is placed in a separate group Gd+1. Finally, n documents are divided into several groups to obtain a group set G={G1,G2,G3,...,Gd,Gd+1}, where G1, G2, G3,...,Gd and Gd+1 represent the first, second, third,...,dth and separate groups respectively. S300 calculates the structural complexity of the documents in the group based on the grouping, selects a representative document according to the structural complexity, and performs structural analysis on the representative document; S400 constructs a cache template for corresponding grouped document detection based on the information representing the document; S500 performs document data quality detection on non-representative documents in the group based on the cache template; When a new document is added to the group, S600 builds and triggers a cache update mechanism, updates the cache template, and completes data quality detection of the document.

2. The resource consumption intelligent monitoring method for document data quality detection according to claim 1 is characterized by: The specific steps of extracting metadata information and structural feature information of a document in S100 are as follows: S101 collects the documents for quality inspection and constructs the original document set D={d1,d2,d3,...,dn}, where d1,d2,d3,... and dn represent the first, second, third,... and nth documents respectively. For the xth document, x∈[1,n], the metadata information of the document is extracted by the document parser and recorded as Yx. The metadata extraction result is represented by the set Xy={Yx1,Yx2,Yx3,...,Yxm}, where Yx1,Yx2, Yx3,... and Yxm represent the first, second, third,... and mth metadata information extracted from the xth document respectively. The structural feature information Tx of the xth document is extracted by parsing the document content. The extraction result of the structural feature is expressed as a set Xt={Tx1,Tx2,Tx3,...,Txb}, where Tx1,Tx2,Tx3,... and Txb represent the first, second, third,... and bth structural feature information extracted from the xth document respectively; S102 combines the metadata information and structural feature information of the x-th document to form an information description of the x-th document, expressed as dx*={Xy,Xt}, and constructs a document set D^={d1*,d2*,d3*,...,dn*} represented by the document information, where d1*, d2*, d3*,... and dn* respectively represent the first, second, third,... and n-th documents containing the document information.

3. The resource consumption intelligent monitoring method for document data quality detection according to claim 2 is characterized by: The specific steps of performing structural analysis on the representative document in S300 are: S301 calculates the structural complexity Ci of the i-th document in the e-th group, e∈[1,d], i represents any document in the e-th group, and the calculation formula is: ， Where ω1, ω2, ω3, ... and ωb are the weights of each structural feature in the i-th document on the document complexity. The document with the largest structural complexity is selected from the e-th group as the representative document dc of the group. S302 analyzes the overall structure of the representative document in the e-th group through document element recognition, title hierarchy analysis, chapter division and chart positioning to obtain a structural analysis Tc, and then formulates corresponding title detection rules, paragraph format rules, table detection rules and chart detection rules for each structural element of the representative document to construct a detection rule set Rc, and then defines a title content template, paragraph content template, table content template and chart content template for each structural unit in the representative document, and defines title format constraints, paragraph format constraints, table format constraints and chart format constraints to obtain a content constraint template Pc.

4. The resource consumption intelligent monitoring method for document data quality detection according to claim 3 is characterized by: The specific steps of constructing a cache template for corresponding group document detection in S400 are: S401 divides the representative documents in the e-th group into several structural units Lj according to structural analysis, and constructs a structural unit set L={Lj1,Lj2,Lj3,...,Ljw}, where Lj1,Lj2,Lj3,... and Ljw represent the first, second, third,... and w-th structural units respectively. For the a-th structural unit in the set L, a∈[1,w], it contains the corresponding detection rule set Rca and content constraint template Pca; S402 stores the information of w structural units as a cache template Md, expressed as Md={(Lj1,Tc1,Rc1,Pc1),(Lj2,Tc2,Rc2,Pc2),(Lj3,Tc3,Rc3,Pc3),...,(Ljw,Tcw,Rcw,Pcw)}, where (Lj1,Tc1,Rc1,Pc1),(Lj2,Tc2,Rc2,Pc2),(Lj3,Tc3,Rc3,Pc3),... and (Ljw,Tcw,Rcw,Pcw) respectively represent the first, second, third,... and wth structural units and the cache template contents corresponding to the structural units, and the cache template Md is used as the cache template corresponding to the eth group.

5. The resource consumption intelligent monitoring method for document data quality detection according to claim 4 is characterized by: The specific steps of performing document data quality detection on the non-representative documents in the group in S500 are: S501: For other non-representative documents in the e-th group, the non-representative documents are divided into a number of structural units based on structural analysis. For each structural unit in the non-representative document, if there is a structural unit in the non-representative document that corresponds to the structural unit in the cache template, the detection rule set and content constraint template of the corresponding structural unit in the cache template are directly reused; S502 When the structural unit in the non-representative document does not exist in the cache template, the corresponding structural unit is marked as a new structural unit, and a temporary detection rule set and content constraint template are generated as a temporary cache, added to the cache template, and a temporary cache survival time threshold ξ is set. When the temporary cache detection is not triggered within the threshold time, the temporary cache is deleted.

6. The resource consumption intelligent monitoring method for document data quality detection according to claim 5 is characterized by: The specific steps for building and triggering the cache update mechanism in S600 are: S601: For the newly added document requiring data quality inspection, calculate the similarity with the representative documents in each group, select the group to which the representative document with the maximum similarity belongs, and add the newly added document to the group; S602 For the e-th group, when a new document is added, the threshold value of the number of new documents in the group is set to Y. When the number of new documents added to the e-th group exceeds the threshold, the structural complexity of the Y new documents in the e-th group is calculated and compared with the structural complexity of the representative document dc in the group. When the structural complexity of the new document is less than or equal to the structural complexity of the representative document dc, the cache template constructed by the representative document is used to perform data quality detection on the new document. When the structural complexity of the new document is greater than the structural complexity of the representative document dc, the new document is used as the new representative document dc*, the new representative document is divided into structural units, and a new cache template Md* is constructed as the cache template of the e-th group. The cache template Md is replaced and the data quality detection is performed on the new document.

7. A resource consumption intelligent monitoring system for document data quality detection, applied to a resource consumption intelligent monitoring method for document data quality detection according to any one of claims 1 to 6, characterized in that: The system includes a feature extraction module, a document grouping module, a parsing module, a cache template building module, a document detection module and a cache update module; The feature extraction module is used to extract metadata information and structural feature information of the document for data quality detection; The document grouping module is used to normalize the metadata information and structural feature information of the documents and group the documents; The parsing module is used to calculate the structural complexity of the documents in the group, select representative documents according to the structural complexity, and perform structural analysis on the representative documents; The cache template construction module is used to construct a cache template for corresponding group document detection based on the information representing the document; The document detection module is used to perform document data quality detection on non-representative documents in the group based on the cache template; The cache update module is used to build and trigger a cache update mechanism when a new document is added to a group.

8. The resource consumption intelligent monitoring system for document data quality detection according to claim 7 is characterized in that: The parsing module includes a complexity calculation unit and a representative document identification unit; The complexity calculation unit is used to calculate the structural complexity of the documents in the group; The representative document identification unit is used to identify the document with the highest structural complexity in the group as the representative document.

Images