Data inventory method based on multi-source material analysis and rule evaluation

Abstract

The present application relates to the technical field of data governance, and discloses a data inventory method based on multi-source material analysis and rule evaluation, which comprises the following steps: after establishing an inventory task, analyzing metadata, scripts, images, business documents, interfaces and running log materials, forming a unified fragment table and standardizing fragment names and texts; writing an abnormal name mark and delaying processing for the fragments that hit the abnormal name rule, and writing a business field mark for the fragments that hit the target business field family rule; grouping objects according to the correspondence of entity names, script references, interface parameters, synonymous expressions and fields; generating transaction window embedded values according to the page submission time, data table writing time, interface sending time and interface timeout length, and then combining the abnormal name mark and the business field mark to generate object retention points, and dividing the object group into a core inventory layer, a secondary retention layer or a redundant trace layer, and outputting a data inventory list with source backtracking information.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, and more specifically, to a data inventory method based on multi-source material analysis and rule evaluation. Background Technology

[0002] In manufacturing execution, quality traceability, warehouse management, and interface integration systems, data governance personnel typically need to aggregate database tables, fields, scripts, interface documentation, page descriptions, and runtime logs into a single inventory task. In real-world environments, business data often first enters the application service via a page, then is written to the database, and subsequently synchronized to downstream systems through a gateway or interface. This path simultaneously leaves access logs, database audit records, interface call records, and timeout configurations. Traditional inventory processes tend to focus on static metadata processing, often filtering and categorizing data according to table name rules, field names, document descriptions, reference counts, or asset catalog levels. Temporary tables, historical tables, system tables, and backup tables are not necessarily invalid objects in real projects; some tables may serve as intermediate storage or historical records after data transfer, quality inspection, warehousing, or equipment acquisition. If the inventory task does not validate the three types of runtime records—page, database, and interface—within the same request channel, it becomes difficult to determine whether a low-frequency table is involved in production data flow. In this case, deletion strategies may exclude necessary data carriers from the catalog, leading to manual checks for subsequent data lineage, interface mapping, standard mapping, and version comparisons. Therefore, an inventory processing method is needed that is oriented towards multi-source materials and computer operation records, retaining suspected objects before rule filtering, and determining the list level based on business fields and operation paths after object aggregation. Summary of the Invention

[0003] This invention provides a data inventory method based on multi-source material analysis and rule evaluation, which solves the technical problems mentioned in the background art.

[0004] To achieve the above-mentioned objectives, the present invention provides the following technical solution.

[0005] A data inventory method based on multi-source material analysis and rule evaluation includes establishing an inventory task, analyzing multi-source materials to form a unified fragment table, and also includes: Standardize segment names and text; Write exception name tags to fragments that match exception name rules and retain them until the object retention score is generated to determine the output destination; Write business field tags according to the target business field family rules used to identify work orders, materials, batches, processes, equipment, inspection, and warehousing fields; Aggregate object groups by entity name, script reference, interface parameter correspondence, synonym expression, and relationship of the same target business field family with consistent field name or field description; The transaction window is defined as the interval between the page submission time and the interface issuance time, provided that the interface timeout period is not exceeded. The embedded value of the transaction window is generated based on whether the data table write time is within the transaction window. Fragments without an exception name tag but with a business field tag are identified as first fragment items, and fragments with an exception name tag, a business field tag, and an embedded value in the transaction window are identified as second fragment items. Based on the first fragment item and the second fragment item, the object retention score is generated. The object group with the first fragment item or the second fragment item is assigned to the core inventory layer. The object group without the first fragment item or the second fragment item but with business field markers or transaction window embedded values ​​indicating that it is located inside the transaction window is assigned to the secondary retention layer. The remaining object groups are assigned to the redundant trace layer. The core inventory layer and the secondary retention layer generate a data inventory list with source traceability information.

[0006] Compared with the prior art, the present invention has the following substantial features and significant progress: This invention incorporates page submission time, data table write time, interface issuance time, and interface timeout duration into the data inventory process. This allows the system to determine whether a database write event is located within the actual business call chain based on existing logs. Simultaneously, it postpones the matching results of exception name rules until after object group aggregation and object retention point generation, avoiding premature removal of low-frequency business tables based solely on table name, frequency, or activity. Therefore, the data inventory list can simultaneously retain material sources, log locations, and hierarchical results, reducing the costs of broken directory chains, duplicate scans, and manual backtracking, and improving the stability of cross-source metadata collection, list updates, and management of objects awaiting confirmation. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the processing architecture of the data inventory method based on multi-source material analysis and rule evaluation of the present invention. Detailed Implementation

[0008] The following description is provided in conjunction with the accompanying drawings. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the invention; those skilled in the art can make equivalent substitutions or combinations for the specific implementations.

[0009] This implementation relates to a data inventory method based on multi-source material analysis and rule evaluation. It can be used in manufacturing execution systems, quality traceability systems, warehouse management systems, data governance platforms, data asset management platforms, and other computer systems that require unified analysis and catalog output of heterogeneous industrial information materials. The method targets electronic materials such as database connection information, data dictionaries, ETL scripts, report scripts, system screenshots, specifications, requirements documents, design documents, user manuals, interface documents, construction standard documents, and computer operation logs. During implementation, the processor 100 reads the aforementioned electronic materials and executes data processing steps, enabling the inventory results to reflect the source relationships between running database objects, design-state document objects, standard-state data standards, interfaces, scripts, pages, or system screenshots.

[0010] Specifically, existing data inventory processes often remove objects based on table name, activity level, or frequency of occurrence before cross-source associations are completed. Objects in temporary, historical, system, or backup tables that actually carry production flow data may be prematurely excluded. Therefore, this implementation first writes fragments that match the exception name rule into exception name tags instead of deleting them immediately. After aggregating the object groups, it combines business field tags and embedded values ​​in the transaction window to generate object retention scores. Thus, even if a data-carrying object has an exception name, verifiable retention criteria can be obtained within the transaction windows defined by page submission, data table writing, and interface issuance.

[0011] like Figure 1 As shown, this embodiment can be executed by processor 100. Processor 100 includes a multi-source material access unit 110, a material parsing and standardization unit 120, a unified fragment table 130, a rule marking unit 140, an object group aggregation unit 150, a transaction window determination unit 160, and a hierarchical list output unit 170.

[0012] Furthermore, the detailed implementation method for establishing inventory tasks and receiving materials from multiple sources is as follows: When processor 100 begins executing the data inventory method based on multi-source material analysis and rule evaluation, it first establishes an inventory task. The inventory task can register the target system, data source connection, inventory scope, and collection time, and generate a material number for each piece of material. The material number is used to maintain traceability among the unified fragment table 130, the data inventory list, the redundancy trace list, the minor retention list, and the discrepancy list.

[0013] The multi-source materials include metadata materials, script materials, image materials, business document materials, interface materials, and operation log materials. Metadata materials include database connection information and data dictionaries; script materials include ETL scripts and report scripts; image materials include system screenshots; business document materials include specifications, requirements documents, design documents, user manuals, and construction standard documents; interface materials include interface documents and interface configurations; and operation log materials include computer operation logs.

[0014] When receiving multi-source materials, the multi-source material access unit 110 records the material number, source system, file type, acquisition time, and original storage location. The file type is used to select the material parser, and the acquisition time and original storage location are used to generate source tracing information. For multiple materials under the same source system, the multi-source material access unit 110 can record them separately according to the actual path of the material and the acquisition time, so that the fragments in the unified fragment table 130 can point back to their corresponding electronic materials, and the tracking basis is not lost due to documents with the same name or repeated file exports.

[0015] In an implementable scenario, the target system is a manufacturing execution system. Metadata materials come from a data dictionary exported from the database, script materials come from the ETL script directory and report script directory, image materials come from system screenshots of the process station pages, business documentation materials come from the requirements document and user manual, interface materials come from the interface document and interface configuration, and operation log materials come from page access logs, application request logs, database audit logs, transaction logs, interface gateway logs, and call logs.

[0016] It should be noted that the multi-source material access unit 110 must not manually supplement missing materials. If a certain source system does not provide operation log materials, the subsequent transaction window determination unit 160 can determine that the embedded value of the corresponding segment's transaction window does not have a basis for the transaction window position; if a certain business document material is missing, the material parsing and standardization unit 120 can still form a unified segment table 130 based on other multi-source materials.

[0017] Furthermore, the detailed implementation method for analyzing materials and forming a unified fragment table is as follows: The material analysis and standardization unit 120 selects a material parser based on the document type. Material type analysis includes: extracting database table fields using a database parser, extracting SQL references using a script parser, extracting chapter tables using a document parser, extracting interface parameters using an interface parser, and extracting page controls using an image parser. Database table fields, SQL references, and interface parameters participate in object group aggregation, while chapter tables and page controls are written to a unified fragment table 130 and used as source information for entity names, text descriptions, and business process nodes.

[0018] The database parser can read database system tables, data dictionary export files, or metadata interfaces to obtain existing information such as database name, table name, field name, field description, and field type. The script parser can parse table names, field names, and interface parameter references in SQL statements, and retain the script file path and the location of the script statements. The document parser can parse chapters, tables, and body text descriptions; the interface parser can parse interface paths, request parameters, response parameters, and configuration items; and the image parser can parse page control names in system screenshots.

[0019] Unified Fragment Table 130 is used to store the parsed fragments. Each fragment must at least store the material number, source type, source system, hierarchical path, entity type, entity name, text description, business process node, collection time, page submission time, data table write time, interface issuance time, interface timeout duration, raw material path, and log record location. The hierarchical path and raw material path are used to generate source backtracking information, the entity type is used for object group aggregation, the text description is used to select the master record in the data inventory list, the business process node is used for sorting the data inventory list, and the collection time is used to generate source backtracking information.

[0020] The page submission time is read from the page access log or application request log. The page submission time represents the time recorded in the page or application log. During business processes such as process transit, inspection entry, warehousing confirmation, or equipment acquisition confirmation, the page submission time typically corresponds to the log record of the operator or system service submitting business data. If the same application request channel has a request number, tracing number, session number, thread number, interface call number, log association field, or an existing call chain identifier in the system, then the page submission time, along with other events in the same application request channel, is written to the unified fragment table 130.

[0021] The data table write time is read from the database audit log or transaction log. The data table write time represents the time when the database write event occurred, recorded in the database audit or transaction log. The corresponding data object can be a regular business table, or a table that matches the naming rules for temporary tables, historical tables, system tables, or backup tables. The data table write time must be saved along with the corresponding segment; simply saving the database table name is insufficient. Otherwise, the transaction window determination unit 160 cannot determine whether the database write event occurred between the page commit time and the interface issuance time.

[0022] The API call time is read from the API gateway log or call log. The API call time represents the time the API call event occurred as recorded in the API gateway or call log, and it is used to form the downstream boundary of the transaction window. The API timeout duration is read from the existing configuration of the API client or gateway. The API timeout duration represents the maximum allowed call duration for this API in the existing configuration, used to avoid incorrectly configuring independent logs that are far apart into the same transaction window. If the page commit time, data table write time, API call time, or API timeout duration are missing, they should remain empty and not be manually added.

[0023] Furthermore, the detailed implementation method for standardizing fragment names and text is as follows: Before or after writing to the unified fragment table 130, the material analysis and standardization unit 120 performs standardization processing on fragment names and text. Standardized fragment names and text include unified uppercase and lowercase letters, full-width and half-width characters, and delimiters; segmentation of camelCase and underscore names; normalization of synonyms for work orders, materials, batches, processes, and equipment using the business dictionary preset by the inventory task; and supplementation of candidate descriptions from the data dictionary, interface documentation, and field comments. The supplemented candidate descriptions are written into the unified fragment table 130 and used to select the master record in the data inventory list.

[0024] The standardization of fragment names and text does not change the original material itself, but adds or updates the standardized results that can be used for object group aggregation in the unified fragment table 130. For tmp_station_pass in the script, api / station / pass in the interface documentation, the station confirmation button in the page control, and the station record field in the data dictionary, the material parsing and standardization unit 120 can enable entity names, text descriptions, and business process nodes to participate in the judgment of the same object group while retaining the raw material path and log record location.

[0025] Unifying English capitalization reduces the impact of capitalization differences on SQL references, interface paths, and field name matching; unifying full-width and half-width characters and separators reduces the impact of character input differences on entity name recognition; camelCase and underscore name segmentation is used to extract business roots from English or Pinyin field names; and a business dictionary is used to standardize business expressions such as work orders, materials, batches, processes, and equipment.

[0026] When supplementing candidate descriptions from the data dictionary, interface documentation, and field comments, the material analysis and standardization unit 120 stores the source material number next to each supplemented candidate description. If the same entity name appears in multiple materials, when selecting the master record in the subsequent data inventory list, the segment with more complete text description, clearer business process nodes, or clearer source path can be prioritized. This process, in conjunction with the unified segment table 130 storing fields for text descriptions, business process nodes, and raw material paths, enables the data inventory list to have a verifiable source.

[0027] The constraints on standardized fragment names and texts are that standardization processing is only used to improve the matching capability across source materials and must not cover the original entity name, original text description, raw material path, and log record location. For fragments that hit anomalous name rules, the material parsing and standardization unit 120 must not delete the fragment prematurely due to the standardization result. The fragment needs to enter the rule marking unit 140 and be retained at least until the object retention fraction generation step.

[0028] Furthermore, the detailed implementation of writing exception name tags and business field tags is as follows: Rule marking unit 140 reads fragments from unified fragment table 130 and executes exception name rules. Exception name rules are used to identify empty tables, backup tables, zombie tables, single-field tables, system tables, temporary tables, log tables, and system configuration tables. Writing exception name tags involves retaining fragments that match the temporary table, historical table, system table, or backup table name rules at least until the object retention part generation step. Exception name tags are used... This means that a value of 1 is used when an exception name rule is matched, and a value of 0 is used when an exception name rule is not matched.

[0029] The purpose of the exception name tag is to transform fragments that might otherwise be directly deleted during the rule-making stage into objects for delayed judgment. For database tables whose names contain tmp_, his_, bak_, or system configuration characteristics, traditional processes might exclude the table based on the name. In this embodiment, the rule tagging unit 140 only writes the exception name tag and writes it back to the unified fragment table 130. This approach allows the fragment to still enter the object group aggregation unit 150 and the transaction window judgment unit 160, thereby obtaining a joint evaluation from the business field tag and the transaction window embedded value.

[0030] Rule tagging unit 140 also executes target business field family rules. Writing business field tags includes executing target business field family rules and identifying a fragment as a fragment with a business field tag when the fragment matches any of the following business fields: work order number, material code, process code, equipment code, batch number, inspection report number, inspection result, quality item, or receiving report number. Business field tags are used... This means that the value is 1 when the target business field family rule is matched, and 0 when the target business field family rule is not matched.

[0031] The purpose of the target business field family rules is to confirm whether a fragment contains content related to the target business object. Simply stating the page submission time, data table write time, and API send time only indicates the sequence of events and cannot prove that the data content belongs to the target business object. Business field markers supplement the content evidence from fields such as work order number, material code, process code, equipment code, batch number, inspection slip number, inspection result, quality item, or receiving slip number. Therefore, an exception name object will only form a second fragment item if it has a business field marker and the embedded value in the transaction window is located inside the transaction window.

[0032] It should be noted that fragments that match the rule for the exception name are not deleted in rule tagging unit 140; fragments that do not match the rule for the target business field family are not manually tagged; and fragments that cannot be confirmed from the field name or field description whether they match the rule for the target business field family are kept in the "missed" state and are hierarchically categorized based on the embedded values ​​in the subsequent transaction window and object group information.

[0033] Further detailed implementation of object group aggregation is as follows: The object group aggregation unit 150 is used to aggregate fragments that point to the same business entity into object groups. Object groups are aggregated according to entity name, script reference, interface parameter correspondence, synonym expression, and relationships of the same target business field family with consistent field names or field descriptions.

[0034] When aggregating by entity name, the object group aggregation unit 150 can use identical entity names as a highly deterministic basis. For example, if `tmp_station_pass` in the data dictionary and `tmp_station_pass` in the database audit log have the exact same entity name, they can be considered fragments of the same data carrier object. When aggregating by script reference, the object group aggregation unit 150 can associate SQL references in ETL scripts or report scripts with database table fields in the data dictionary. When aggregating by interface parameter mapping, the object group aggregation unit 150 can establish mappings between request parameters, response parameters, and database table fields in the interface documentation.

[0035] When aggregating by synonyms, the object group aggregation unit 150 uses the business dictionary preset by the inventory task to group synonyms for work orders, materials, batches, processes, and equipment into the same business entity. When aggregating by relationships with the same target business field family and consistent field names or descriptions, the object group aggregation unit 150 requires fragments to have the same target business field family and consistent field names or descriptions to avoid mistakenly merging different object groups simply because they are similar in business domain. This restriction enables object group aggregation to support the generation of subsequent object retention scores without causing incorrect propagation of embedded values ​​across objects within the transaction window.

[0036] After the object group is established, the source mapping between the runtime database object, the design-state document object, and the standard-state data standard is saved. In one embodiment, tmp_station_pass is used as the runtime database object, the station pass confirmation page is used as the page source, api / station / pass is used as the interface source, the station pass confirmation section in the requirements document is used as the design-state document object, and the process station pass field requirements in the construction standard document are used as the standard-state data standard. The object group aggregation unit 150 merges the above fragments into the same object group, so that the hierarchical list output unit 170 can select the master record from the object group and retain the associated source.

[0037] It should be noted that the aggregation criteria must come from existing entity names, text descriptions, business process nodes, source information, script references, interface parameter correspondences, synonyms, target business field families, field names, or field descriptions in the unified fragment table 130. Business entities that do not appear in the multi-source materials must not be manually added. If two fragments are only close in collection time but lack the above-mentioned aggregation criteria, the object group aggregation unit 150 must not merge them into the same object group.

[0038] Furthermore, the detailed implementation methods for transaction window determination and embedded values ​​within the transaction window are as follows: The transaction window determination unit 160 is used to generate an embedded value for the transaction window based on the page commit time, the data table write time, the interface issuance time, and the interface timeout duration. The transaction window is bounded by the interval from the page commit time to the interface issuance time, which does not exceed the interface timeout duration. When the data table write time is within this interval, it indicates that the database write event has a transaction window position between the page commit time and the interface issuance time.

[0039] The process of generating embedded values ​​within a transaction window includes: determining whether the data table write time is later than the page submission time but earlier than the interface issuance time within the same application request channel, and determining whether the interval between the page submission time and the interface issuance time does not exceed the interface timeout duration; generating embedded values ​​within a transaction window indicating that the database write event is located within the transaction window when the determination result is true, and generating embedded values ​​within a transaction window indicating that there is no basis for the transaction window position when the determination result is false or any of the aforementioned times or the interface timeout duration is missing.

[0040] Embedded values ​​in transaction window The following formula can be used for calculation: in, Indicates the first The page submission time corresponding to each segment is taken from the page access log or application request log. Indicates the first The data table write time corresponding to each segment is taken from the database audit log or transaction log; Indicates the first The time when the interface corresponding to each segment is issued is taken from the interface gateway log or call log; Indicates the first The timeout duration for each segment is taken from the existing configuration of the interface client or gateway. An indicator operator that takes a value of 1 if the condition is true and 0 otherwise; This indicates the value embedded within the transaction window.

[0041] The conditions in the above formula Simultaneously, three relationships are defined. First, the data table write time is later than the page commit time, excluding the case where the database write event occurs earlier than the page commit event. Second, the data table write time is earlier than the interface issuance time, excluding the case where the database write event occurs after the interface issuance time. Third, the interval between the page commit time and the interface issuance time does not exceed the interface timeout duration, excluding the case where independent logs that are far apart are incorrectly configured into the same transaction window.

[0042] In the formula This requires that a database write event have both an upstream interval after the page commit time and a downstream interval before the interface issue time. The formula... This is used to normalize the total interval from the page submission time to the interface issuance time, thereby reducing the impact of differences in throughput speed and absolute log time between different servers on the judgment result. A coefficient of 4 ensures that the value embedded within the transaction window can reach its upper limit of 1 when the database write event is in the middle of the transaction window, and is greater than 0 when strictly within the transaction window.

[0043] For segments where the sequence is incorrect, the interval between the page submission time and the interface issuance time exceeds the interface timeout duration, the page submission time is missing, the data table write time is missing, the interface issuance time is missing, or the interface timeout duration is missing, the transaction window determination unit 160 generates an embedded value indicating that there is no basis for the transaction window position. When implemented numerically, the embedded value indicating that there is no basis for the transaction window position can be recorded as 0; when implemented in a state, it can be recorded as a state indicating that there is no basis for the transaction window position. Regardless of whether a numerical or state implementation is used, missing page submission times, data table write times, interface issuance times, or interface timeout durations must not be manually added.

[0044] Furthermore, the detailed implementation of object retention and hierarchical list output is as follows: The hierarchical inventory output unit 170 is used to generate object retention scores based on the first and second fragment items, and to divide the inventory into a core inventory layer, a secondary retention layer, and a redundant trace layer based on the object retention scores. The first fragment item is a fragment without an exception name tag but with a business field tag. The second fragment item is a fragment with an exception name tag, a business field tag, and an embedded value in the transaction window indicating that it is located inside the transaction window. The first fragment item reflects the business field basis of ordinary name objects, while the second fragment item reflects the business field basis and transaction window position basis of exception name objects.

[0045] Object retention is divided into object-level hierarchical state values. In the numerical implementation, object retention can be... It is represented and generated according to the following formula: in, Represents a group of objects. Indicates the first The fragment belongs to the object group. , Indicates the exception name marker, Indicates a business field marker. This indicates the value embedded within the transaction window. This indicates the object's retained score. In this formula, fragments without an exception name tag but with a business field tag are... and Fragment items with a value of 1 are formed at that time; fragments with an exception name marker, a business field marker, and embedded values ​​within the transaction window indicate that they are located inside the transaction window. and Formed by Given a fragment item; the object group takes the maximum value of all fragment items.

[0046] The purpose of taking the maximum value of an object group is to avoid amplifying the frequency of fragments or weakening low-frequency objects. If a low-frequency temporary table appears only once, but it has a business field marker and the transaction window embedded value indicates that it is inside the transaction window, then the second fragment item can enable the object group to obtain the object retention score; if an object group contains a large number of duplicate fragments but none of them have a business field marker and the transaction window embedded value does not indicate that they are inside the transaction window, then the duplicate fragments will not accumulate in number to form the basis of the core inventory layer.

[0047] In the state implementation method, when there is a first fragment item or a second fragment item in the same object group, the object retention score is determined as the core inventory layer state; when there is no first fragment item or second fragment item in the same object group but there is a fragment with a business field mark or a fragment with a transaction window embedded value indicating that the database write event is located inside the transaction window, the object retention score is determined as the secondary retention layer state; when there is no fragment with a business field mark in the same object group and no fragment with a transaction window embedded value indicating that the database write event is located inside the transaction window, the object retention score is determined as the redundant trace layer state.

[0048] The core inventory layer is used to store object groups with strong retention criteria. Object groups without exception name tags but with business field tags can enter the core inventory layer through the first fragment item; object groups with exception name tags must also have business field tags and the embedded value in the transaction window must be located inside the transaction window in order to enter the core inventory layer through the second fragment item.

[0049] The secondary retention layer is used to store groups of objects that still require confirmation but should not be automatically deleted in this round. When an object group does not have a first fragment item or a second fragment item, but has a business field marker or a transaction window embedded value indicating that it is inside a transaction window, the object group is placed in the secondary retention layer. Objects in the secondary retention layer can be added to the confirmation list for subsequent manual review or comparison with the next version of the catalog. This setting can reduce the risk of catalog chain breakage and avoid directly classifying objects with only a single piece of evidence into the redundant trace layer.

[0050] The redundant trace layer is used to store object groups that do not have business field tags and whose embedded values ​​in the transaction window do not represent objects located inside the transaction window. Objects in this layer can still be written to the redundant trace list and retain source traceability information, but they are not output as core objects in the data inventory list. Through the layering of the core inventory layer, secondary retention layer, and redundant trace layer, the layered list output unit 170 can simultaneously provide retained objects, objects to be confirmed, and redundant trace objects.

[0051] The hierarchical inventory output unit 170 generates a data inventory list with source traceability information from the core inventory layer and the secondary retention layer. When generating the data inventory list, the hierarchical inventory output unit 170 selects the fragment with the highest object retention score and the most complete text description from the object groups of the core inventory layer and the secondary retention layer as the master record. Database tables serve as the data carrier objects, and corresponding entities in interfaces, scripts, pages, and documents are written to the associated source. The data inventory list may include the business object name, data carrier object, source system, business process node, raw material path, object level, object retention score, exception name marker, business field marker, transaction window embedded value, and update time.

[0052] The hierarchical list output unit 170 can also compare the data inventory list with the previous version directory to generate a difference list. During the comparison, records with the same name and the same data carrying object can be merged and multiple source paths can be retained; records with the same name but different data carrying objects are saved separately and associated; newly added, modified, deleted, and pending confirmation objects are recorded separately. Secondary retained layer objects are added to the pending confirmation list and are not automatically deleted in this round.

[0053] As an optional implementation, the data carrier object does not match the exception name rule, but the fragment matches the target business field family rule. For example, a regular business table contains work order number, material code, and process code, and the rule tagging unit 140 writes business field tags. And the exception name is marked Even if the fragment temporarily lacks the page submission time, data table write time, interface issuance time, or interface timeout duration, the hierarchical list output unit 170 can still identify it as the first fragment item. When the first fragment item exists in the object group, the object retention score is identified as the core inventory layer state, and the core inventory layer generates the data inventory list.

[0054] As an alternative implementation, the data-bearing object hits an exception name rule, but it is actually located in the production data path. For example, tmp_station_pass hits a temporary table name rule, and rule tagging unit 140 writes an exception name tag. This segment contains the work order number, process code, equipment code, and transit time; the rule tagging unit 140 writes the business field tag. The transaction window determination unit 160 reads the page commit time, data table write time, interface issue time, and interface timeout duration, and generates an embedded value representing the transaction window within the transaction window. At this time, the hierarchical list output unit 170 determines this fragment as the second fragment item, and the object group enters the core inventory layer.

[0055] As an optional implementation, an object group may lack a first fragment item and a second fragment item, but it may contain fragments marked with business fields. For example, a historical table may match an exception name rule and have a business field mark, but the runtime log material is missing, causing the embedded value in the transaction window to not be indicated as being inside the transaction window. This object group will not enter the core inventory layer, but because it has a business field mark, the hierarchical list output unit 170 will classify it into the secondary retention layer. Objects in the secondary retention layer will enter the pending confirmation list and will not be automatically deleted in this round.

[0056] As an optional implementation, an object group may lack a first fragment item and a second fragment item, but may have a transaction window embedded value indicating that a database write event is located within a transaction window. For example, a log table may lack explicit business field descriptions, and the business field markers may not be matched, but the page submission time, data table write time, interface issuance time, and interface timeout duration are complete. The transaction window determination unit 160 generates a transaction window embedded value indicating that the event is located within a transaction window. This object group is also classified into a secondary reserved layer for subsequent review of field descriptions or business documentation sources.

[0057] As an optional implementation, if any one of the following is missing: page submission time, data table write time, interface issuance time, or interface timeout duration, the transaction window determination unit 160 does not manually fill in the missing information. Instead, it generates an embedded value indicating that the transaction window lacks a positional basis. If the object group also lacks a business field marker, the hierarchical list output unit 170 classifies it into the redundancy trace layer; if the object group has a business field marker, the hierarchical list output unit 170 classifies it into the secondary retention layer. This process ensures that missing logs do not cause error core retention and also prevents objects with business field markers from being directly deleted.

[0058] As an optional implementation, the data table write time is earlier than the page commit time, or the data table write time is later than the interface issuance time, or the interval between the page commit time and the interface issuance time exceeds the interface timeout duration. The transaction window determination unit 160 generates embedded values ​​for transaction windows that do not have a transaction window position basis. If an object group is close to the interface material only because of similar names, but does not meet the event order in the same application request channel, the object group will not enter the core inventory layer based on time proximity, thereby reducing erroneous retention.

[0059] As an optional implementation, if a fragment in an object group does not match the target business field family rule, and the embedded value in the transaction window does not indicate that the database write event is located inside the transaction window, the hierarchical list output unit 170 classifies the object group into the redundant trace layer and outputs a redundant trace list. The redundant trace list retains the material number, raw material path, and log record location, so that subsequent audits can still trace the source of the object group, but the object group does not enter the core inventory layer or the secondary retention layer.

[0060] It should be noted that this implementation method can be executed by a task scheduling module in a server, personal computer, cloud computing instance, or data governance platform, or it can be executed as an inventory task process in an existing data asset management system, metadata management system, or data governance platform.

[0061] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A data inventory method based on multi-source material analysis and rule evaluation, comprising establishing an inventory task and analyzing multi-source materials to form a unified fragment table, characterized in that, Also includes: Standardize segment names and text; Write exception name tags to fragments that match exception name rules and retain them until the object retention score is generated to determine the output destination; Write business field tags according to the target business field family rules used to identify work orders, materials, batches, processes, equipment, inspection, and warehousing fields; Aggregate object groups by entity name, script reference, interface parameter correspondence, synonym expression, and relationship of the same target business field family with consistent field name or field description; The transaction window is defined as the interval between the page submission time and the interface issuance time, provided that the interface timeout period is not exceeded. The embedded value of the transaction window is generated based on whether the data table write time is within the transaction window. Fragments without an exception name tag but with a business field tag are identified as first fragment items, and fragments with an exception name tag, a business field tag, and an embedded value in the transaction window are identified as second fragment items. Based on the first fragment item and the second fragment item, the object retention score is generated. The object group with the first fragment item or the second fragment item is assigned to the core inventory layer. The object group without the first fragment item or the second fragment item but with business field markers or transaction window embedded values ​​indicating that it is located inside the transaction window is assigned to the secondary retention layer. The remaining object groups are assigned to the redundant trace layer. The core inventory layer and the secondary retention layer generate a data inventory list with source traceability information.

2. The data inventory method based on multi-source material analysis and rule evaluation according to claim 1, characterized in that, Multi-source materials include metadata materials, script materials, image materials, business document materials, interface materials, and operation log materials; Metadata materials include database connection information and data dictionaries; script materials include ETL scripts and report scripts; image materials include system screenshots; business document materials include specifications, requirements documents, design documents, user manuals, and construction standard documents; interface materials include interface documents and interface configurations; and operation log materials include computer operation logs. When creating an inventory task, record the material number, source system, file type, collection time, and original storage location. The file type is used to select the material parser, and the collection time and original storage location are used to generate source traceability information.

3. The data inventory method based on multi-source material analysis and rule evaluation according to claim 2, characterized in that, Material type analysis includes: extracting database table fields through a database parser, extracting SQL references through a script parser, extracting chapter tables through a document parser, extracting interface parameters through an interface parser, and extracting page controls through an image parser; database table fields, SQL references, and interface parameters participate in object group aggregation, while chapter tables and page controls are written into a unified fragment table and used as source information for entity names, text descriptions, and business process nodes.

4. The data inventory method based on multi-source material analysis and rule evaluation according to claim 2, characterized in that, The page submission time is read from the page access log or application request log; the data table write time is read from the database audit log or transaction log; the interface issuance time is read from the interface gateway log or call log; and the interface timeout duration is read from the existing configuration of the interface client or gateway. If the page submission time, data table write time, interface issuance time, or interface timeout duration are missing, keep them as null values.

5. The data inventory method based on multi-source material analysis and rule evaluation according to claim 3, characterized in that, Each fragment in the unified fragment table stores the material number, source type, source system, hierarchical path, entity type, entity name, text description, business process node, collection time, page submission time, data table write time, interface issuance time, interface timeout duration, raw material path, and log record location. The hierarchical path and raw material path are used to generate source traceability information, the entity type is used for object group aggregation, the text description is used to select the master record in the data inventory list, the business process node is used for sorting the data inventory list, and the collection time is used to generate source traceability information.

6. The data inventory method based on multi-source material analysis and rule evaluation according to claim 5, characterized in that, Name and text standardization includes unifying English capitalization, full-width and half-width characters and separators, segmenting camelCase and underscore names, using the business dictionary preset by the inventory task to unify synonyms for work orders, materials, batches, processes and equipment, and supplementing candidate descriptions from the data dictionary, interface documentation and field comments. The supplemented candidate descriptions are written into a unified fragment table and used to select the master record in the data inventory list.

7. The data inventory method based on multi-source material analysis and rule evaluation according to claim 1, characterized in that, Anomaly naming rules are used to identify empty tables, backup tables, zombie tables, single-field tables, system tables, temporary tables, log tables, and system configuration tables; writing anomaly name tags includes retaining fragments that hit the temporary table, historical table, system table, or backup table name rules at least until the object retention part generation step.

8. The data inventory method based on multi-source material analysis and rule evaluation according to claim 1, characterized in that, Writing business field tags involves executing the target business field family rules and identifying the fragment as a fragment with a business field tag when the fragment matches any of the following business fields: work order number, material code, process code, equipment code, batch number, inspection order number, inspection result, quality item, or warehouse receipt number.

9. The data inventory method based on multi-source material analysis and rule evaluation according to claim 1, characterized in that, The process of generating embedded values ​​within a transaction window includes: determining whether the data table write time is later than the page submission time but earlier than the interface issuance time within the same application request channel, and determining whether the interval between the page submission time and the interface issuance time does not exceed the interface timeout duration; generating embedded values ​​within a transaction window indicating that the database write event is located within the transaction window when the determination result is true, and generating embedded values ​​within a transaction window indicating that there is no basis for the transaction window position when the determination result is false or any of the aforementioned times or the interface timeout duration is missing.

10. The data inventory method based on multi-source material analysis and rule evaluation according to claim 1, characterized in that, Object retention is divided into object-level hierarchical status values; when there is a first fragment item or a second fragment item in the same object group, the object retention is determined to be in the core inventory layer status; when there is no first fragment item or second fragment item in the same object group but there are fragments with business field tags or fragments whose transaction window embedded values ​​indicate that the database write event is located inside the transaction window, the object retention is determined to be in the secondary retention layer status; when there are no fragments with business field tags in the same object group and no fragments whose transaction window embedded values ​​indicate that the database write event is located inside the transaction window, the object retention is determined to be in the redundancy trace layer status; the data inventory list is compared with the previous version catalog to generate a difference list.

Images