An intelligent engine and construction method for data assetization governance

Abstract

This invention relates to an intelligent engine and its construction method for data asset governance, relating to the field of big data processing. The engine includes: collecting real-time data update records, historical business data, and target data archiving dates; statistically analyzing the amount of data to be registered and organized, the amount of data to be pre-archived, and the amount of unchanged data based on the real-time data update records and historical business data, and generating pre-archiving tasks in conjunction with the target data archiving dates; obtaining an archived dataset based on the pre-archiving tasks; obtaining a data digital digest and a global hash value based on the archived dataset; generating a data storage certificate by combining the data digital digest and the global hash value; integrating the archived dataset and the data storage certificate to obtain an archived database conversion package; and pushing the archived database conversion package to an electronic archive room to complete the data asset transfer and generate an electronic transfer form. This invention improves the accuracy and completeness of archived data.

Description

Technical Field

[0001] This invention relates to the field of big data processing, and in particular to an intelligent engine and its construction method for data asset governance. Background Technology

[0002] The intelligent engine for data asset governance refers to an intelligent system that integrates functions such as data collection, processing, archiving, evidence storage, and transfer, and is used to help enterprises or institutions transform business data into manageable, auditable, and traceable data assets.

[0003] Currently, data governance technology has gradually upgraded from traditional manual and rule-based governance to intelligent and automated governance. In recent years, artificial intelligence technology has been deeply integrated with data governance, realizing the automation of the entire data governance process.

[0004] However, existing automated data governance solutions mostly focus on data cleaning, classification, and quality improvement. Data screening before archiving relies on fixed time windows or manually specified ranges, lacking refined statistics on the dynamic changes in data, which reduces the accuracy and completeness of archived data and needs improvement. Summary of the Invention

[0005] To improve the accuracy and completeness of archived data, this invention provides an intelligent engine and its construction method for data asset governance, employing the following technical solution: An intelligent engine and its construction method for data asset governance, comprising: Collect real-time data update records, historical business data, and the archiving date of target data input by the user from the business system; Based on real-time data update records and historical business data, the amount of data to be registered and organized, the amount of data to be pre-archived and processed, and the amount of data that has not changed are statistically analyzed. The pre-archiving task is generated by combining the amount of data to be registered and organized, the amount of data to be pre-archived, the amount of data that has not changed, and the target data archiving date. Based on the pre-archiving task to obtain the archived dataset; The data digital digest and global hash value are calculated based on the archived dataset; Combine data digital digests with global hash values ​​to generate data storage certificates; Integrate archived datasets and data storage certificates to obtain an archived database conversion package; Push the archived database conversion package to the electronic archive room to complete the data asset transfer and generate an electronic transfer form.

[0006] By adopting the above technical solution, the scope of archiving is refined by statistically analyzing the amount of data to be registered, pre-archived, and unchanged, and dynamically generating pre-archiving tasks based on the archiving date. Then, by calculating the data digital digest and global hash value, evidence storage certificates are generated. The archived dataset and evidence storage certificates are integrated to form a conversion package and pushed to the electronic archive room to generate an electronic transfer form. This achieves full automation of the process from data screening and evidence storage to transfer, improving the accuracy and completeness of data archiving.

[0007] Optionally, a method for determining the archived dataset may also be included: Generate a unique task code based on the pre-archived task; Data processing is performed on the data to be archived corresponding to the pre-archived task based on the unique task code, and a structured execution log is generated synchronously. The execution status of the data processing task is determined based on the structured execution log; If the task execution status is normal, the data processing result of this task code will be output as the archived dataset. If the task execution status is abnormal, then collect user input commands; An exception handling method is generated based on user input instructions, and the archived dataset is determined in response to the exception handling method.

[0008] By adopting the above technical solution, a unique task code is generated based on the pre-archived task, and a structured execution log is generated synchronously. The decision to output the archived dataset is made based on the task execution status. If an execution error occurs, the data is edited or repackaged according to user instructions, ensuring the correctness and traceability of the archived dataset and improving the fault tolerance capability in abnormal scenarios.

[0009] Optionally, the exception handling method includes: When the user inputs an edit command, the data to be archived under that task code is located based on the edit command. Correct the located data to be archived according to the editing instructions; Reprocess the corrected data to be archived and output the archived dataset; When the user inputs a repackaging command, a new unique task code is generated based on the repackaging command. Bind the new unique task code to the original data to be archived, re-execute the data processing, and output the new archived dataset.

[0010] By adopting the above technical solution, when the user selects the edit command, abnormal data can be located and corrected before reprocessing the output; when the user selects the repackaging command, a new task code is generated and the entire process is re-executed, avoiding the contamination of subsequent archiving by dirty data associated with the original task code, and ensuring the flexibility of data cleaning and the purity of archived data.

[0011] Optionally, methods for tracing and querying archived data are also included: Collect user-initiated requests for archived data tracing; Based on the archived data tracing request, task coding information is obtained; Locate the corresponding archived dataset based on the task coding information; Combining unique task codes and structured execution logs to reconstruct the entire data processing flow nodes; A visual traceability chain is generated based on the information of all nodes in the process and the corresponding archived datasets; The visual traceability link is fed back to the user's front end and the traceability display is completed.

[0012] By adopting the above technical solution, the corresponding archived dataset can be located and the entire data processing process nodes can be restored based on the task code in the traceability request initiated by the user. A visual traceability link can be generated, realizing transparent traceability of data from the original record to the archive transfer, and improving the auditability of data assets and the efficiency of problem investigation.

[0013] Optionally, methods for verifying the consistency of the transferred content may also be included: Verification trigger command after data asset transfer is completed; Based on the verification trigger command, retrieve the corresponding electronic handover form and associated task code; Match the associated archived dataset with the associated data storage certificate based on the associated task code; The data content, number of entries, and hash information of the associated archived dataset, associated data storage certificate, and corresponding electronic transfer form are compared to obtain the consistency verification results.

[0014] By adopting the above technical solution, after the handover is completed, the data content, number of entries, and hash information of the archived dataset, evidence certificate, and handover form are compared according to the electronic handover form and task code. The consistency verification is automatically completed to ensure that the handover content has not been tampered with or omitted, thereby enhancing the credibility of data asset handover.

[0015] Optional, vulnerability detection methods may also be included: Collect information on all data assets that have been transferred. Determine a globally unique task code based on data asset information; The original archived dataset, data authenticity certificate, and data transfer process record are determined based on the globally unique task code. The original archived dataset, data authenticity certificates, and data transfer process records are combined to generate a set of files to be inspected; The consistency and integrity of the globally unique task code across all files are uniformly tested based on the set of files to be tested. When the globally unique task code is detected to be lost, tampered with, or mismatched in any file, a corresponding risk warning is generated. Based on the correlation risk warning, the frequency of occurrence of different types of risks is statistically analyzed and risk analysis reports are generated.

[0016] By adopting the above technical solutions, the consistency and integrity of the globally unique task code in the archived dataset, evidence certificate and transfer record are uniformly tested for all transferred data assets. Risks such as loss, tampering or mismatch are identified, the frequency of occurrence of various risks is statistically analyzed and risk analysis reports are generated, so as to realize the proactive discovery and quantitative assessment of data asset vulnerability.

[0017] Optionally, methods for generating DNA watermarks for data assets are also included: The original association key is generated based on globally unique task encoding; Independent key fragments are obtained by splitting the original associated key; Generate DNA barcode sequences based on independent key fragments; Generate digital invisible watermarks based on DNA barcode sequences; The digital invisible watermark is embedded into the original archived dataset, the data authenticity certificate, and the data transfer process record.

[0018] By adopting the above technical solution, the original associated key is generated based on the globally unique task encoding, split into independent key fragments and converted into DNA barcode sequences, and then a digital invisible watermark is generated. These watermarks are embedded into the archived dataset, evidence storage certificate and transfer record, respectively, to provide cross-file same-source association identifier for data assets, and enhance anti-tampering capabilities and multi-file collaborative verification capabilities.

[0019] Optionally, a distributed association self-recovery method based on DNA watermarking is also included: The warning task code is determined based on the associated risk warning; Based on the early warning task code, early warning archive data, early warning evidence documents, and early warning transfer documents are obtained; Extract the corresponding hidden watermark based on the early warning archive data, early warning evidence documents, and early warning transfer documents; The corresponding barcode sequence is reconstructed based on the corresponding invisible watermark; The original key fragment is reconstructed based on the corresponding barcode sequence; Integrity verification is performed on the original key fragments to filter out valid key fragments, and the number of valid key fragments is collected. When the number of valid fragments is not less than the preset valid number threshold, the original associated key is reconstructed based on the valid key fragments; Re-establish the relationships between all data assets based on the original association key.

[0020] By adopting the above technical solution, when a risk of association is detected, the barcode sequence and key fragments are extracted from the warning data to restore the hidden watermark. Valid fragments are screened through integrity verification. When the number of valid fragments reaches a threshold, the original association key is reconstructed, and the association relationship between data assets is re-established. This enables data assets to self-recover in a distributed environment and improves the system's disaster recovery and damage resistance capabilities.

[0021] Optionally, a DNA watermark-based end-to-end traceability and verification method is also included: The original key barcode sequence is retrieved based on the early warning task encoding; Extract on-site watermark barcodes based on archived early warning data, early warning evidence documents, and early warning handover documents; The differences between the barcodes are obtained by comparing the original key barcode sequence with the on-site watermark barcode. Based on differences in barcode comparison, the data tampering process and the abnormal entity can be located; Based on the data tampering process and the abnormal entities, a complete traceability of the data flow can be established; Based on the complete flow and traceability trajectory, a full-chain security traceability certificate for data assets is formed.

[0022] By adopting the above technical solution, the original key barcode sequence is retrieved based on the early warning task code and compared with the on-site watermark barcode extracted from archived data, evidence files and transfer documents. The difference is used to locate the data tampering link and the abnormal subject, sort out the complete flow traceability trajectory and form a full-link security traceability certificate, thereby improving data reliability.

[0023] Optional, a watermark-based tiered encryption protection method is also included: The watermark security protection level is determined based on the early warning task code; The corresponding encryption key is determined based on the watermark security level; The DNA barcode sequence is encrypted in layers according to the corresponding encryption key.

[0024] In summary, this application includes at least one of the following beneficial technical effects: 1. By statistically analyzing the amount of data to be registered, pre-archived, and unchanged, and dynamically generating pre-archived tasks based on the archiving date, the scope of archiving can be finely filtered. Then, by calculating the data digital digest and global hash value, evidence storage certificates are generated. The archived dataset and evidence storage certificates are integrated to form a conversion package and pushed to the electronic archives room to generate an electronic transfer form. This achieves full automation of the process from data screening and evidence storage to transfer, improving the accuracy and completeness of data archiving. 2. Based on the globally unique task encoding, the original associated key is generated, split into independent key fragments and converted into DNA barcode sequences, and then a digital invisible watermark is generated. These watermarks are embedded into the archived dataset, evidence storage certificate and transfer record, respectively, to provide cross-file same-source association identifier for data assets, enhance anti-tampering capabilities and multi-file collaborative verification capabilities; 3. When a risk of association is detected, the hidden watermark is extracted from the warning data to restore the barcode sequence and key fragment. Valid fragments are filtered through integrity verification. When the number of valid fragments reaches a threshold, the original association key is reconstructed to re-establish the association relationship between data assets, realize the self-recovery of data assets in a distributed environment, and improve the disaster recovery and damage resistance of the system. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the methodology and construction method of an intelligent engine for data asset governance. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0027] Reference Figure 1 This application discloses an intelligent engine and its construction method for data asset governance, including the following steps: S10: Collect real-time data update records, historical business data, and the archiving date of target data entered by the user from the business system.

[0028] Real-time data update records refer to the timestamps and operation types of data row insertion, update, and deletion operations recorded in the business system database. These records are captured in real-time by the intelligent engine through the database log listening interface.

[0029] Historical business data refers to the complete collection of all business data tables and their associated data stored in the business system up to the current moment. This historical business data is periodically extracted by the intelligent engine through the database connector.

[0030] The target data archiving date refers to the cutoff date entered by the user through the interactive interface, which defines the time range of the data to be archived.

[0031] S11: Based on real-time data update records and historical business data, to calculate the amount of data to be registered and organized, the amount of data to be pre-archived and processed, and the amount of data that has not changed.

[0032] The amount of data to be registered and organized refers to the total number of newly generated data entries in the business system that have not yet been marked as organized since the completion of the last archiving task. The amount of data to be registered and organized is obtained by scanning the real-time data update records and counting the number of records whose operation type is "new" and whose timestamp is later than the completion time of the last archiving task.

[0033] The pre-archived data volume refers to the total number of data entries in historical business data that meet preset archiving conditions but have not yet been archived. The pre-archived data volume is obtained by filtering and counting historical business data using preset archiving rules. These preset archiving rules are set in advance by those skilled in the art based on the requirements of business data lifecycle management.

[0034] The unchanged data quantity refers to the total number of data entries in historical business data whose content has not been modified or deleted by any business operations between the completion of the last archiving task and the target data archiving date. The unchanged data quantity is obtained by comparing the data content of historical business data on the target data archiving date with the data content at the time of completion of the last archiving task, and counting the number of records whose content is completely identical.

[0035] S12: Combine the amount of data to be registered and organized, the amount of data to be pre-archived and processed, the amount of data that has not changed, and the target data archiving date to generate a pre-archiving task.

[0036] A pre-archiving task refers to a task entity to be executed, which includes a task identifier, data volume statistics, and a data deadline. A pre-archiving task is assembled by using the amount of data to be registered and processed, the amount of data to be pre-archived, and the amount of data that has not changed as data volume statistics fields, and the target data archiving date as the data deadline field, and generating a unique task identifier.

[0037] S13: Obtain the archived dataset based on the pre-archiving task.

[0038] An archived dataset refers to a collection of data to be archived, organized according to a preset data structure, generated after the pre-archiving task is performed. The specific method for determining the archived dataset will be explained in detail in subsequent sections S20 to S25, and will not be repeated here.

[0039] S14: Calculate the data digital digest and global hash value based on the archived dataset.

[0040] A data digital digest is a fixed-length unique feature value calculated from the content of each data record in an archived dataset. The data digital digest is calculated by applying a preset digest algorithm to the field content of each data record. The preset digest algorithm is selected in advance by those skilled in the art based on data security requirements.

[0041] A global hash value is a unique identifier used to identify the overall integrity of an archived dataset, calculated from all its contents. The global hash value is calculated by applying a predefined hash algorithm to the binary content of the archived dataset files. The specific implementation methods of the digest and hash algorithms are common knowledge in the field.

[0042] S15: Combine the data digital digest with the global hash value to generate a data storage certificate.

[0043] A data preservation certificate is a structured proof document that contains integrity verification information and unique identification information of an archived dataset. A data preservation certificate is assembled by storing a digital digest of the data in a certificate file as a record-by-record format, using a global hash value as the identifier field of the certificate file, and appending a generation timestamp.

[0044] S16: Integrate archived datasets and data storage credentials to obtain an archived database conversion package.

[0045] An archived database conversion package refers to a single data packet that conforms to the receiving format requirements of an electronic archives room, formed by merging and packaging archived dataset files and data preservation certificate files. The archived database conversion package is obtained by placing the archived dataset files and data preservation certificate files in the same file directory and compressing and packaging them according to a preset packaging format. The packaging format is predetermined by those skilled in the art based on the interface specifications of the electronic archives room.

[0046] S17: Push the archived database conversion package to the electronic archive room, complete the data asset transfer and generate an electronic transfer form.

[0047] An electronic archive is an independent storage and management system used to preserve archived data assets for the long term and provide data query and auditing functions.

[0048] The intelligent engine calls the data receiving interface of the electronic archives and pushes the archived database conversion package to the electronic archives via network transmission. After the electronic archives completes the reception, it returns a receipt confirmation. Based on this confirmation, the intelligent engine generates an electronic transfer document. The electronic transfer document is a structured credential file that records the transfer time, the amount of data transferred, the global hash value, and the identification information of both parties involved in the data asset transfer.

[0049] It also includes methods for determining archived datasets: S20: Generate a unique task code based on the pre-archived task.

[0050] A unique task code is a string identifier used to globally and uniquely identify this pre-archived task. The unique task code is calculated by combining the current system timestamp, the business system number, and a random sequence number, then applying a preset encoding generation algorithm. This encoding generation algorithm is pre-defined by those skilled in the art based on the uniqueness requirements of the task identifier.

[0051] S21: Perform data processing on the data to be archived corresponding to the pre-archived task based on the unique task code, and generate a structured execution log simultaneously.

[0052] Data to be archived refers to the set of data extracted from historical business data and awaiting processing, as pointed to by the data volume statistics field in the pre-archiving task. Data to be archived is obtained by parsing the data range description field of the pre-archiving task and combining it with a unique task code to pinpoint the data batch corresponding to this task.

[0053] Structured execution logs are detailed log data recorded in a preset field format, describing the entire data processing process. Structured execution logs include fields for task start time, task end time, amount of data processed, amount of data unchanged, and operation status at each processing stage. The structured execution log is generated by the intelligent engine, which writes it to the log database stage by stage during data processing. The preset field format is pre-defined by those skilled in the art based on task monitoring and auditing requirements.

[0054] S22: Determine the execution status of the data processing task based on the structured execution log.

[0055] Task execution status refers to the status indicator reflecting whether each stage of this data processing task has been successfully completed. The task execution status is obtained by parsing the operation status fields of each processing stage in the structured execution log. When all operation status fields of all stages are successful, the task execution status is normal; when any operation status field of any stage is a failure, the task execution status is abnormal.

[0056] S23: If the task execution status is normal, output the data processing result of the task code bound to this task as the archived dataset.

[0057] An archived dataset refers to a collection of data to be archived, organized according to a preset data structure, output when the task execution status is normal. An archived dataset is obtained by binding the data processing results with a unique task code.

[0058] S24: If the task execution status is abnormal, collect user input commands.

[0059] User input commands refer to the operation instructions issued by users through the interactive interface for abnormal tasks. User input commands include editing commands and repackaging commands. Editing commands are used to correct the content of the data to be archived corresponding to the abnormal task, while repackaging commands are used to trigger a reprocessing process for the original data to be archived.

[0060] S25: Generate an exception handling method based on user input instructions, and determine the archived dataset in response to the exception handling method.

[0061] An exception handling method refers to a specific processing flow determined based on the type of user input instructions, used to handle exception tasks and ultimately output an archived dataset.

[0062] Upon receiving a user input instruction, the type of the instruction needs to be determined. If the user input instruction is an edit instruction, steps S30 to S32 are executed. If the user input instruction is a repackaging instruction, steps S33 to S34 are executed. The specific editing and repackaging processes will be described in detail in subsequent embodiments and will not be repeated here.

[0063] Exception handling methods include: S30: When the user inputs an edit command, locate the data to be archived under the task code based on the edit command.

[0064] Editing commands are user-issued instructions via an interactive interface to modify specific data entries in an abnormal task. Editing commands include data location information and correction content information. The data location information specifies the data entry to be corrected, and the correction content information specifies the corrected field value.

[0065] The data to be archived under this task code refers to the set of original data to be processed extracted in this pre-archiving task and bound to the unique task code generated by S20. This data to be archived is obtained by querying the task data association table using the unique task code.

[0066] The task data association table is a data table used to record the mapping relationship between the unique task code and the storage paths of each data file generated by this task, as well as the original data storage location to be archived. The task data association table is created when S21 data processing starts and includes a task code field, a file type field, and a storage path field. The task code field records the unique task code generated in S20, the file type field identifies the category of the associated file, and the storage path field records the physical location of the associated file in the storage system.

[0067] When the user inputs an edit command, the data to be archived under that task code must first be located for subsequent steps.

[0068] S31: Correct the located data to be archived according to the editing instructions.

[0069] The intelligent engine parses the data location information in the editing instructions and locates the target data entry in the data to be archived. It then parses the correction information in the editing instructions and replaces the original value of the corresponding field in the target data entry with the field value specified in the correction information, thus completing the data correction.

[0070] S32: Re-execute data processing on the corrected data to be archived and output the archived dataset.

[0071] The intelligent engine re-executes the S21 data processing flow on the corrected data to be archived, generating a new structured execution log. When the task execution status is normal, the processing result is output as the archived dataset. The archived dataset refers to the collection of data to be archived, organized according to a preset data structure after correction.

[0072] S33: When the user inputs a repackaging instruction, generate a new unique task code based on the repackaging instruction.

[0073] A repackaging command is an operation command issued by a user through an interactive interface to trigger a complete reprocessing of the original data to be archived.

[0074] In response to the repackaging command, the intelligent engine calls the unique task code generation method of S20, using the current system timestamp, business system number, and new random sequence number as input to generate a new unique task code.

[0075] When the user inputs a repackaging command, a new unique task code must first be generated for subsequent steps.

[0076] S34: Bind the new unique task code to the original data to be archived, re-execute the data processing, and output the new archived dataset.

[0077] The intelligent engine establishes a binding relationship between the new unique task code and the original data to be archived in the task data association table. It then re-executes the S21 data processing flow on the original data to be archived, generating a structured execution log corresponding to the new task code. When the task execution status is normal, a new archived dataset is output. The new archived dataset refers to the collection of data to be archived, bound to the new unique task code and organized according to a preset data structure.

[0078] It also includes methods for tracing and querying archived data: S40: Collect user-initiated requests for tracing archived data.

[0079] An archived data tracing request refers to a user's request, entered through an interactive interface, to query the entire process of a specified archived data asset from its generation to its transfer. The archived data tracing request includes the database conversion package file number or archive number corresponding to the archived data to be queried. Users initiate this request by entering the database conversion package file number or archive number in the search box on the tracing query interface and clicking the query button.

[0080] S41: Obtain task coding information based on archived data tracing requests.

[0081] Task code information refers to a unique task code associated with the database conversion package file number or file number in the archived data traceability request. Task code information is obtained as follows: the intelligent engine parses the database conversion package file number or file number in the archived data traceability request and queries the handover record table for the unique task code bound to that file number or file number. The handover record table is automatically written by the intelligent engine each time a data asset handover is completed, recording the correspondence between the database conversion package file number and the unique task code.

[0082] S42: Locate the corresponding archived dataset based on the task coding information.

[0083] The corresponding archived dataset refers to the set of archived data associated with the pre-archived task identified by the task coding information. The corresponding archived dataset is located in the following way: the intelligent engine uses the task coding information as the query condition, retrieves the storage path information of the archived dataset bound to the task coding in the task data association table, and accesses the archived data storage area to obtain the archived dataset based on the storage path information.

[0084] S43: Combine unique task codes and structured execution logs to reconstruct the entire data processing flow nodes.

[0085] The data processing workflow nodes refer to a complete set of workflow nodes describing the execution status of each stage of the data to be archived, from extraction, processing, packaging to transfer. The data processing workflow nodes are reconstructed in the following way: The intelligent engine uses a unique task code as the query condition to retrieve the complete structured execution log records under that task code from the log database; according to the timestamp field in the structured execution log, the operation status, amount of data processed, and amount of data unchanged in each processing stage are arranged in chronological order to construct the data processing workflow node sequence.

[0086] S44: Generate a visual traceability link based on the full-process node information and the corresponding archived dataset.

[0087] A visual traceability chain refers to a visual data chain that is displayed in a graphical form, containing all nodes of the data processing process and the corresponding data status of each node.

[0088] The intelligent engine arranges the entire process node information according to a timeline, labeling each node with the operation name, execution status, amount of data processed, and execution time for that stage; it displays the basic information of the corresponding archived dataset as the starting node of the link; and it connects the nodes with directed lines to form a complete visual link from data extraction to data transfer. The specific graphical rendering method is common knowledge in the field.

[0089] S45: Feed back the visualized traceability link to the user's front end and complete the traceability display.

[0090] The intelligent engine sends the generated visual traceability data to the user's front-end interface in a preset data format. The user's front-end interface parses this data format and graphically presents the entire data processing flow nodes and detailed information for each node in the traceability query results area. When the user clicks on any flow node, the interface displays the structured execution log details corresponding to that node.

[0091] It also includes methods for verifying the consistency of the transferred content: S50: Verification trigger instruction after data asset transfer is completed.

[0092] A verification trigger command is a confirmation signal returned to the intelligent engine by the electronic archives after successfully receiving the archived database conversion package and completing data verification. This signal confirms the completion of the transfer and triggers a consistency verification. The verification trigger command includes the database conversion package file number and the electronic archives' signature field.

[0093] S51: Retrieve the corresponding electronic handover document and associated task code based on the verification trigger command.

[0094] The corresponding electronic transfer document is a structured credential file that matches the database conversion package file number in the verification trigger instruction and records the information for this data asset transfer. The corresponding electronic transfer document can be retrieved from the transfer record table using the database conversion package file number.

[0095] The associated task code is a unique identifier for a pre-archived task associated with a corresponding electronic handover document. The associated task code is obtained by parsing the task identifier field stored in the corresponding electronic handover document.

[0096] S52: Match the associated archived dataset with the associated data storage certificate based on the associated task code.

[0097] The associated archive dataset refers to the collection of archived data generated in this pre-archiving task and associated with the associated task code. The associated archive dataset is obtained by querying its storage path in the task data association table using the associated task code.

[0098] A data storage certificate associated with a task is a data storage certificate file generated during this pre-archiving task and is bound to the associated task code. The associated data storage certificate is obtained by querying its storage path in the task data association table using the associated task code.

[0099] S53: Compare the data content, number of entries, and hash information of the associated archived dataset, associated data storage certificate, and corresponding electronic transfer form to obtain consistency verification results.

[0100] The intelligent engine extracts the total number of data entries and the content of each data record from the associated archived dataset, extracts the data digital digest and global hash value of the record from the associated data storage certificate, and extracts the amount of transferred data and global hash value from the corresponding electronic transfer document.

[0101] Data content comparison is achieved by recalculating the digital digest of each data record in the associated archive dataset and matching it line by line with the corresponding digital digest of the data in the associated data preservation certificate. Item quantity comparison is achieved by numerically comparing the total number of data items in the associated archive dataset with the amount of transferred data recorded in the corresponding electronic transfer document. Hash information comparison is achieved by recalculating the global hash value of the associated archive dataset and performing a three-way match with the global hash value in the associated data preservation certificate and the corresponding electronic transfer document.

[0102] The consistency verification result is passed when the data content, the number of entries, and the hash information are consistent. If any one of these three aspects is inconsistent, the consistency verification result is failed, and the type and location information of the inconsistent item are output.

[0103] It also includes vulnerability detection methods: S60: Collect information on all data assets that have been transferred.

[0104] Data asset information refers to the collection of all data asset records in the intelligent engine's handover record table that are in a "handover completed" state. Data asset information includes the database transformation package file number and the corresponding globally unique task code for each record.

[0105] S61: Determine a globally unique task code based on data asset information.

[0106] A globally unique task code is a unique identifier corresponding to the data asset to be tested and used throughout the entire process of data extraction, processing, packaging, and transfer. The globally unique task code is extracted record by record from the data asset information.

[0107] S62: Use the globally unique task code to identify the original archived dataset, data authenticity certificate, and data transfer process record.

[0108] The raw archive dataset refers to the collection of archived data files generated during the execution of a pre-archiving task and bound to a globally unique task code. The raw archive dataset is obtained by querying the storage path in the task data association table using the globally unique task code.

[0109] A data authenticity certificate is a structured proof document that is bound to a globally unique task code and contains integrity verification information and unique identification information of the archived dataset. The data authenticity certificate is obtained by querying the storage path in the task data association table using the globally unique task code.

[0110] A data transfer process record is an electronic transfer document file that is bound to a globally unique task code and records key information about this data asset transfer operation. The data transfer process record is obtained by querying the storage path in the transfer record table using the globally unique task code.

[0111] S63: Combine the original archived dataset, data authenticity certificates, and data transfer process records to generate a set of files to be tested.

[0112] The set of files to be tested refers to a group of files formed by combining the original archived dataset corresponding to the same globally unique task code, the data authenticity certificate, and the data transfer process record. The set of files to be tested is generated by aggregating the storage paths of the three files into a single test task record.

[0113] S64: Based on the set of files to be tested, uniformly test the consistency and integrity of the globally unique task code in each file.

[0114] The intelligent engine traverses each file group in the set of files to be tested, reading the metadata area of ​​the original archived dataset, the identifier field of the data authenticity certificate, and the task code field of the data transfer process record sheet, and extracting the globally unique task code value recorded in each file. The three extracted globally unique task code values ​​are compared pairwise, and the globally unique task code field in each file is checked to see if it is empty or if there is an abnormal encoding format.

[0115] S65: When the globally unique task code is detected to be lost, tampered with, or mismatched in any file, generate a corresponding type of associated risk warning.

[0116] Associated risk warning refers to alarm information used to identify abnormal types of data asset association relationships detected.

[0117] When an empty globally unique task code field is detected, a risk warning for code loss is generated. When inconsistent globally unique task code values ​​extracted from different files are detected, a risk warning for code tampering is generated. When the globally unique task code format does not conform to the preset encoding rules, a risk warning for identifier mismatch is generated. The preset encoding rules are set in advance by those skilled in the art and will not be elaborated here.

[0118] S66: Based on associated risk warnings, statistically analyze the frequency of occurrence of different types of risks and generate risk analysis reports.

[0119] The intelligent engine calculates the frequency of each of the three risk warning types—code loss, code tampering, and identifier mismatch—in this detection. It then calculates the percentage of each risk warning type out of the total number of file groups detected. The statistical results are organized according to risk type, frequency of occurrence, percentage, and corresponding globally unique task code list to generate a risk analysis report.

[0120] It also includes methods for generating DNA watermarks for data assets: S70: Generate the original association key based on globally unique task encoding.

[0121] The original association key refers to the root key data used to establish relationships between data assets, generated by encrypting the globally unique task code. The original association key is calculated by applying a preset national cryptographic algorithm to the globally unique task code. This preset encryption algorithm is selected in advance by those skilled in the art based on data security level requirements.

[0122] S71: Obtain independent key fragments by splitting the original associated key.

[0123] Independent key segments refer to multiple key sub-segments formed by dividing the original associated key according to preset splitting rules.

[0124] The preset splitting rule refers to dividing the byte sequence of the original associated key into equal-length segments according to a preset number of segments, padding any trailing bytes that are not of equal length with zero values, so that each independent key segment is of equal length. The preset number of segments is determined by those skilled in the art based on the redundancy requirements of watermark embedding and the minimum number of segments required for key recovery.

[0125] S72: Generate DNA barcode sequences based on independent key fragments.

[0126] DNA barcode sequences are sequences of four characters: adenine, guanine, cytosine, and thymine, obtained by converting independent key fragments according to base mapping rules.

[0127] The binary data of each independent key segment is divided into groups of two bits. Each group of binary values ​​is mapped to a corresponding base character according to a pre-defined base mapping table. All base characters are then concatenated sequentially to form a DNA barcode sequence. The base mapping table defines a one-to-one correspondence between binary values ​​and four types of base characters, and is pre-defined by those skilled in the art.

[0128] S73: Generate digital invisible watermarks based on DNA barcode sequences.

[0129] Digital invisible watermarking refers to the identification information formed by embedding DNA barcode sequences into carrier data, which cannot be directly perceived by the human eye or conventional tools.

[0130] The DNA barcode sequence is converted into binary watermark data. This binary watermark data is then superimposed onto the frequency domain transform coefficients of the carrier data according to a preset embedding strength parameter. An inverse transform is then performed to obtain the watermarked carrier data. The carrier data refers to the binary content of the original archived dataset file, data authenticity certificate file, or data transfer process record file. The preset embedding strength parameter is pre-set by those skilled in the art based on the balance requirements of watermark imperceptibility and robustness. The specific implementation methods of the frequency domain transform and inverse transform are common knowledge in the field.

[0131] S74: Embed the digital invisible watermark into the original archived dataset, the data authenticity certificate, and the data transfer process record.

[0132] The intelligent engine uses the binary content of the original archived dataset file, the data authenticity certificate file, and the data transfer process record file as carrier data. It performs the S73 digital invisible watermark generation process on each carrier data, embedding a digital invisible watermark converted from the same DNA barcode sequence into each of the three files. This results in a watermarked original archived dataset, a watermarked data authenticity certificate, and a watermarked data transfer process record. The three files contain the same DNA barcode sequence information, forming a data asset association identifier based on the DNA watermark.

[0133] It also includes a distributed association self-recovery method based on DNA watermarking: S80: Determine the warning task code based on the associated risk warning.

[0134] The early warning task code refers to the globally unique task code corresponding to a data asset identified as having a missing, tampered, or mismatched code in the associated risk early warning. The early warning task code is extracted from the associated risk early warning information.

[0135] S81: Obtain early warning archive data, early warning evidence documents, and early warning transfer documents based on the early warning task code.

[0136] Early warning archive data refers to the original archived dataset file that is bound to the early warning task code and has an embedded digital invisible watermark. Early warning archive data is obtained by querying the storage path in the task data association table using the early warning task code.

[0137] Early warning evidence files refer to data authenticity verification documents that are bound to the early warning task code and have embedded digital invisible watermarks. Early warning evidence files are obtained by querying the storage path in the task data association table using the early warning task code.

[0138] An early warning handover document is a data handover process record file that is linked to the early warning task code and has an embedded digital watermark. The early warning handover document is retrieved by querying the storage path in the handover record table using the early warning task code.

[0139] S82: Extract the corresponding hidden watermark based on the early warning archive data, early warning evidence documents, and early warning transfer documents.

[0140] Corresponding invisible watermarks refer to digital invisible watermark data embedded in early warning archive data, early warning evidence files, and early warning transfer documents, respectively. Corresponding invisible watermarks are obtained by performing the same frequency domain transformation on each file as described in S73, and extracting the watermark binary data from the preset embedding positions in the transformation coefficients. The specific watermark extraction algorithm is common knowledge in the field and will not be elaborated here.

[0141] The preset embedding position refers to the specific set of coefficient indices selected in the frequency domain transform coefficients of the carrier data during the watermark embedding process to hide the binary data of the watermark. The preset embedding position is determined during the S73 watermark embedding process and is synchronized with the watermark extraction stage.

[0142] S83: Reconstruct the corresponding barcode sequence based on the corresponding invisible watermark.

[0143] The corresponding barcode sequence refers to the DNA sequence string composed of four characters—adenine, guanine, cytosine, and thymine—reconstructed from the binary data of the corresponding invisible watermark. The extracted invisible watermark binary data is decoded using the reverse process of the S73 embedding method to obtain the corresponding barcode sequence. The specific decoding method is common knowledge in the field.

[0144] S84: Reconstruct the original key fragment based on the corresponding barcode sequence.

[0145] The original key fragment refers to the key sub-fragment that constitutes the original associated key, which is restored from the corresponding barcode sequence. The base characters in the corresponding barcode sequence are converted into a pair of binary data according to the inverse mapping relationship of the base mapping table in S72. All binary data are then concatenated sequentially to obtain the original key fragment.

[0146] S85: Perform integrity verification based on the original key fragment to filter valid key fragments, and collect the number of valid key fragments.

[0147] A valid key fragment refers to an original key fragment that has passed integrity verification and whose data format and length meet preset requirements. The process checks whether the byte length of each original key fragment matches the preset fragment length in S71, and whether its data format conforms to the key fragment encoding standard. Key fragments that pass the checks are marked as valid key fragments. The number of valid fragments refers to the total number of valid key fragments obtained after filtering.

[0148] S86: When the number of valid fragments is not less than the preset valid number threshold, the original associated key is reconstructed based on the valid key fragments.

[0149] The effective quantity threshold refers to the minimum number of valid fragments required to determine whether there is sufficient information to reconstruct the original associated key. This threshold is determined by those skilled in the art based on the principle of rounding down to half the preset fragment number in S71. When the number of valid fragments is not less than the effective quantity threshold, the intelligent engine sorts and splices the valid key fragments according to their fragment numbers to recover the complete original associated key.

[0150] S87: Re-establish the association between all data assets based on the original association key.

[0151] The intelligent engine uses the original association key obtained from the reconstruction as an index to re-establish the binding relationship between the early warning archive data, early warning evidence files, and early warning transfer documents and the early warning task code in the task data association table and transfer record table, thereby restoring the distributed association between data assets.

[0152] It also includes a DNA watermark-based end-to-end traceability and verification method: S90: Retrieve the original key barcode sequence based on the early warning task encoding.

[0153] The original key barcode sequence refers to the DNA sequence string composed of four characters: adenine, guanine, cytosine, and thymine, generated in step S72 and bound to the warning task code. The original key barcode sequence is retrieved from the watermark sequence storage area by querying the watermark barcode sequence storage record in the task data association table using the warning task code.

[0154] The watermark sequence storage area is a dedicated storage area used for persistently storing DNA barcode sequences and their binding relationship with unique task codes. The watermark sequence storage area is created during the initialization of the intelligent engine and contains a task code field and a barcode sequence field. The task code field records the unique task code generated by S20, and the barcode sequence field records the generated DNA barcode sequence content.

[0155] S91: Extract the on-site watermark barcode based on the early warning archive data, early warning evidence documents, and early warning handover documents.

[0156] On-site watermark barcodes refer to DNA barcode sequences extracted and restored from the current version of early warning archive data, early warning evidence documents, and early warning transfer documents. The on-site watermark barcode is obtained by performing the watermark extraction process (S82) on each document to obtain the corresponding invisible watermark, and then converting the invisible watermark into a DNA barcode sequence according to the barcode sequence restoration process (S83).

[0157] S92: Obtain the barcode comparison difference based on the original key barcode sequence and the on-site watermark barcode.

[0158] Barcode comparison discrepancies refer to the recorded information on the positions and types of mismatched bases after comparing the original key barcode sequence with each on-site watermark barcode position by base. Using the original key barcode sequence as the baseline sequence, a positional comparison is performed with the on-site watermark barcodes extracted from the early warning archive data, early warning evidence documents, and early warning transfer documents. The position index and mismatch type of each base mismatch are recorded.

[0159] S93: Locate the data tampering process and abnormal entities based on the differences in barcode comparison.

[0160] The data tampering stage refers to the specific processing stage in the entire data extraction, processing, packaging, or transfer process determined by the base fragment at the location index of the barcode comparison difference. This stage maps the comparison difference location to the base interval corresponding to the independent key fragment of S72. Based on the base interval where the difference location falls, the damaged key fragment is determined, and then the fragment number of the damaged key fragment is mapped to a specific stage in the entire data processing process.

[0161] An anomaly subject refers to the identifier of the responsible party that caused the abnormal change in file content during the data tampering process. The anomaly subject is obtained by querying the operation subject field recorded in the structured execution log corresponding to the data tampering process.

[0162] S94: Based on the data tampering process and the abnormal entities, a complete flow traceability trajectory is established.

[0163] A complete data transfer traceability track refers to a complete record chain presented in timeline form, showing the operating entities, operation times, operation content, and the integrity status of the corresponding on-site watermark barcodes at each stage of the entire process from data extraction to transfer. The complete data transfer traceability track is organized as follows: The intelligent engine uses the early warning task code as an index to retrieve the data processing node information of S43 throughout the entire process. Data tampering stages are marked as abnormal nodes on the timeline, and the abnormal entity information is associated with the corresponding node. Simultaneously, the consistency status of the on-site watermark barcode and the original key barcode sequence of the corresponding file at each node is marked, forming a complete data transfer traceability track that includes both normal and abnormal nodes.

[0164] S95: Based on the complete flow traceability trajectory, form a data asset full-link security traceability certificate.

[0165] A data asset end-to-end security traceability certificate is a structured traceability report file that includes a complete flow traceability trajectory, barcode comparison difference details, data tampering location results, and information on the abnormal entity. The data asset end-to-end security traceability certificate is created by organizing the complete flow traceability trajectory in a graphical node link format, attaching barcode comparison differences in a difference detail table, filling in the data tampering stage and abnormal entity as conclusion fields, and then encapsulating it with a generation timestamp and certificate issuance identifier.

[0166] It also includes watermark-based tiered encryption protection methods: SA0: Determine the watermark security protection level based on the early warning task code.

[0167] The watermark security protection level refers to a graded identifier used to guide the watermark encryption strength, determined based on the type and severity of associated risk warnings. The watermark security protection level is obtained by matching the associated risk warning types generated in S65 with preset level mapping rules. The preset level mapping rules define the first protection level as the code loss risk, the second protection level as the code tampering risk, and the third protection level as the identifier mismatch risk. The encryption strength of the first, second, and third protection levels increases progressively. The preset level mapping rules are pre-set by those skilled in the art based on the degree of impact of different types of risks on data asset security.

[0168] SA1: Determine the corresponding encryption key based on the watermark security protection level.

[0169] The corresponding encryption key refers to the key data used to encrypt the DNA barcode sequence, which matches the watermark security protection level. When the watermark security protection level is Level 1, a preset basic encryption key is used; when the watermark security protection level is Level 2, a preset enhanced encryption key is used; and when the watermark security protection level is Level 3, a preset high-strength encryption key is used. The preset basic encryption key, enhanced encryption key, and high-strength encryption key are retrieved from the key management system by those skilled in the art according to the security strength required for each protection level.

[0170] SA2: The DNA barcode sequence is encrypted in layers according to the corresponding encryption key.

[0171] The intelligent engine uses the DNA barcode sequence generated by S72 as the data to be encrypted, applies the corresponding encryption key determined by SA1, and encrypts it according to the encryption algorithm corresponding to the watermark security protection level. The first protection level uses single-layer symmetric encryption, the second protection level uses double-layer nested encryption, and the third protection level uses three-layer chain encryption. The encrypted DNA barcode sequence replaces the original sequence and enters the digital invisible watermark generation process in S73. The specific implementation methods of symmetric encryption, nested encryption, and chain encryption are common knowledge in the field.

[0172] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An intelligent engine and build method for data assetization governance characterized in that, include: Collect real-time data update records, historical business data, and the archiving date of target data input by the user from the business system; Based on real-time data update records and historical business data, the amount of data to be registered and organized, the amount of data to be pre-archived and processed, and the amount of data that has not changed are statistically analyzed. The pre-archiving task is generated by combining the amount of data to be registered and organized, the amount of data to be pre-archived, the amount of data that has not changed, and the target data archiving date. Based on the pre-archiving task to obtain the archived dataset; The data digital digest and global hash value are calculated based on the archived dataset; Combine data digital digests with global hash values ​​to generate data storage certificates; Integrate archived datasets and data storage certificates to obtain an archived database conversion package; Push the archived database conversion package to the electronic archive room to complete the data asset transfer and generate an electronic transfer form.

2. The intelligent engine and build method for data assetization governance as claimed in claim 1 wherein, It also includes methods for determining archived datasets: Generate a unique task code based on the pre-archived task; Data processing is performed on the data to be archived corresponding to the pre-archived task based on the unique task code, and a structured execution log is generated synchronously. The execution status of the data processing task is determined based on the structured execution log; If the task execution status is normal, the data processing result of this task code will be output as the archived dataset. If the task execution status is abnormal, then collect user input commands; An exception handling method is generated based on user input instructions, and the archived dataset is determined in response to the exception handling method.

3. The intelligent engine and build method for data assetization governance as claimed in claim 2 wherein, The exception handling method includes: When the user inputs an edit command, the data to be archived under that task code is located based on the edit command. Correct the located data to be archived according to the editing instructions; Reprocess the corrected data to be archived and output the archived dataset; When the user inputs a repackaging command, a new unique task code is generated based on the repackaging command. Bind the new unique task code to the original data to be archived, re-execute the data processing, and output the new archived dataset.

4. The intelligent engine and build method for data assetization governance as claimed in claim 2 wherein, It also includes methods for tracing and querying archived data: Collect user-initiated requests for archived data tracing; Based on the archived data tracing request, task coding information is obtained; Locate the corresponding archived dataset based on the task coding information; Combining unique task codes and structured execution logs to reconstruct the entire data processing flow nodes; A visual traceability chain is generated based on the information of all nodes in the process and the corresponding archived datasets; The visual traceability link is fed back to the user's front end and the traceability display is completed.

5. The intelligent engine and build method for data assetization governance as claimed in claim 4 wherein, It also includes methods for verifying the consistency of the transferred content: Verification trigger command after data asset transfer is completed; Based on the verification trigger command, retrieve the corresponding electronic handover form and associated task code; Match the associated archived dataset with the associated data storage certificate based on the associated task code; The data content, number of entries, and hash information of the associated archived dataset, associated data storage certificate, and corresponding electronic transfer form are compared to obtain the consistency verification results.

6. The intelligent engine and build method for data assetization governance as claimed in claim 5 wherein, It also includes vulnerability detection methods: Collect information on all data assets that have been transferred. Determine a globally unique task code based on data asset information; The original archived dataset, data authenticity certificate, and data transfer process record are determined based on the globally unique task code. The original archived dataset, data authenticity certificates, and data transfer process records are combined to generate a set of files to be inspected; The consistency and integrity of the globally unique task code across all files are uniformly tested based on the set of files to be tested. When the globally unique task code is detected to be lost, tampered with, or mismatched in any file, a corresponding risk warning is generated. Based on the correlation risk warning, the frequency of occurrence of different types of risks is statistically analyzed and risk analysis reports are generated.

7. The intelligent engine and build method for data assetization governance as claimed in claim 6 wherein, It also includes methods for generating DNA watermarks for data assets: The original association key is generated based on globally unique task encoding; Independent key fragments are obtained by splitting the original associated key; Generate DNA barcode sequences based on independent key fragments; Generate digital invisible watermarks based on DNA barcode sequences; The digital invisible watermark is embedded into the original archived dataset, the data authenticity certificate, and the data transfer process record.

8. The intelligent engine and build method for data assetization governance as claimed in claim 7 wherein, It also includes a distributed association self-recovery method based on DNA watermarking: The warning task code is determined based on the associated risk warning; Based on the early warning task code, early warning archive data, early warning evidence documents, and early warning transfer documents are obtained; Extract the corresponding hidden watermark based on the early warning archive data, early warning evidence documents, and early warning transfer documents; The corresponding barcode sequence is reconstructed based on the corresponding invisible watermark; The original key fragment is reconstructed based on the corresponding barcode sequence; Integrity verification is performed on the original key fragments to filter out valid key fragments, and the number of valid key fragments is collected. When the number of valid fragments is not less than the preset valid number threshold, the original associated key is reconstructed based on the valid key fragments; Re-establish the relationships between all data assets based on the original association key.

9. The intelligent engine and construction method for data asset governance according to claim 8, characterized in that, It also includes a DNA watermark-based end-to-end traceability and verification method: The original key barcode sequence is retrieved based on the early warning task encoding; Extract on-site watermark barcodes based on archived early warning data, early warning evidence documents, and early warning handover documents; The differences between the barcodes are obtained by comparing the original key barcode sequence with the on-site watermark barcode. Based on differences in barcode comparison, the data tampering process and the abnormal entity can be located; Based on the data tampering process and the abnormal entities, a complete traceability of the data flow can be established; Based on the complete flow and traceability trajectory, a full-chain security traceability certificate for data assets is formed.

10. The intelligent engine and construction method for data asset governance according to claim 8, characterized in that, It also includes watermark-based tiered encryption protection methods: The watermark security protection level is determined based on the early warning task code; The corresponding encryption key is determined based on the watermark security level; The DNA barcode sequence is encrypted in layers according to the corresponding encryption key.

Images