Unified hash identification dataset field management method and system with recovery capability

Abstract

The present application belongs to the technical field of big data processing and business intelligence, and particularly relates to a data set field management method and system with unified hash identification and recovery capability, wherein a data set field management operation request initiated by a user is acquired; a data set table and a data set field table stored in a metadata database are constructed; when the operation type is data set creation, data set metadata is saved to the data set table and a self-incremented data set unique identification is generated; if the to-be-created field is a basic field, a hash value is calculated after normalizing the basic field original name to obtain a field unique identification; if the to-be-created field is a derived field, all field names in the normalized SQL expression are replaced with corresponding stored field unique identifications to obtain a field unique identification; the data set field table is stored in, and field automatic recovery, data set replication and downstream data set switching operations are executed. Thus, the problems of high overall operation and maintenance cost, complicated operation and easy errors in the prior art are solved.

Description

Technical Field

[0001] This invention belongs to the field of big data processing and business intelligence technology, specifically relating to a method and system for managing dataset fields with unified hash identifiers and recovery capabilities. Background Technology

[0002] This invention belongs to the field of big data processing and business intelligence (BI) technology, focusing on the full lifecycle management of fields in datasets within BI systems. Existing dataset field management technologies mainly undertake core functions such as data table field identification and allocation, metadata storage, relationship maintenance, dataset replication, and field CRUD operations. They can adapt to the dynamic adjustment needs of datasets brought about by enterprise business iterations, ensuring the normal connection between fields and downstream ETL operations, data reports, visualization dashboards, and data analysis logic. It is a key underlying support technology for enterprises to build a data asset system, implement data-driven decision-making, and maintain the stable flow of data links.

[0003] The existing technology suffers from the following core defects: It does not use a normalized hashing method based on field names to generate unique identifiers for basic fields, nor does it generate stable unique identifiers for derived fields by normalizing the SQL expressions of derived fields, replacing field names, and then calculating hashes. This results in unstable and inconsistent field identifier generation rules. Furthermore, it lacks pre-built dataset tables and dataset field tables adapted for unified hash identifier management, and does not store field identifiers as strings, failing to support persistent management of stable identifiers. When a field is deleted and rebuilt, the original identifier cannot be reused, and there is no automatic field recovery capability, leading to a direct break in downstream dependencies. When a dataset is copied, field identifiers are forcibly changed, and the dependency relationships of derived fields cannot be automatically migrated, requiring manual maintenance of identifier mappings. When downstream businesses switch datasets, each field reference must be modified individually; automatic compatibility cannot be achieved simply by changing the dataset's unique identifier, resulting in high overall maintenance costs, cumbersome operations, and a high risk of errors. Summary of the Invention

[0004] This application provides a method and system for managing dataset fields with unified hash identifiers and recovery capabilities, in order to solve the problems of high overall operation and maintenance costs, cumbersome operation and easy error in the prior art.

[0005] Firstly, the implementation scheme disclosed herein provides a method for managing dataset fields with recoverable unified hash identifiers, including: Retrieve user-initiated requests for dataset field management operations; The dataset field management operation request is parsed to obtain the operation type, target dataset information, and field information to be processed. A dataset table and a dataset field table stored in the metadata database are pre-built. When the operation type is dataset creation, dataset metadata is generated based on the target dataset information, the dataset metadata is saved to the dataset table, and an auto-incrementing unique identifier for the dataset is generated; When the operation type is field creation, if the field to be created is a basic field, the hash value of the original name of the basic field is calculated after normalization to obtain the unique identifier of the field. When the field to be created is a derived field, the original SQL expression of the derived field is normalized, and all field names in the normalized SQL expression are replaced with the corresponding stored unique identifier of the field. The hash value of the replaced expression is calculated to obtain the unique identifier of the field. The unique identifier of the field is associated with the corresponding field metadata and stored in the dataset field table. Based on the unique identifier of the string field in the dataset field table, the automatic field recovery, dataset copying and downstream dataset switching operations are performed.

[0006] Secondly, the implementation scheme disclosed herein provides a unified hash identifier dataset field management system with recovery capabilities, including: The request retrieval module is used to retrieve user-initiated requests for dataset field management operations. The request parsing module is used to parse the dataset field management operation request to obtain the operation type, target dataset information and field information to be processed, and to pre-build the dataset table and dataset field table stored in the metadata database; The dataset creation module is used to generate dataset metadata based on the target dataset information when the operation type is dataset creation, save the dataset metadata to the dataset table, and generate an auto-incrementing unique identifier for the dataset. The field identifier generation module is used to, when the operation type is field creation, if the field to be created is a basic field, calculate the hash value after normalizing the original name of the basic field to obtain the unique identifier of the field; when the field to be created is a derived field, normalize the original SQL expression of the derived field, replace all field names in the normalized SQL expression with the corresponding stored unique identifier of the field, and calculate the hash value of the replaced expression to obtain the unique identifier of the field. The storage operation and maintenance execution module associates the unique identifier of the field with the corresponding field metadata and stores it in the dataset field table. Based on the unique identifier of the string field in the dataset field table, it performs automatic field recovery, dataset copying and downstream dataset switching operations.

[0007] Thirdly, embodiments of this disclosure provide a data transmission device, including: processor; Memory, used to store executable instructions; The processor is used to read executable instructions from memory and execute the executable instructions to implement the unified hash identifier dataset field management method with recovery capability in the first aspect.

[0008] Fourthly, the embodiments of this disclosure provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the unified hash identifier dataset field management method of the first aspect embodiment, which has recoverable capabilities.

[0009] Fifthly, this disclosure provides a computer program product, which includes a computer program that, when executed by a processor, implements the unified hash identifier dataset field management method with recoverability as described in the first aspect of the embodiment.

[0010] Therefore, this application offers the following beneficial effects: By generating unified and unique field hash identifiers using differentiated hash calculation rules for basic and derived fields, and relying on pre-built dataset tables and dataset field tables to complete metadata association and storage, it can standardize field names and derived SQL expression formats, effectively avoiding data errors caused by field name conflicts and expression modifications. Furthermore, it can smoothly achieve automatic field recovery, dataset replication, and downstream dataset switching operations using dedicated hash field identifiers, significantly improving field traceability accuracy and data asset reuse efficiency, strengthening data operation and maintenance fault tolerance, reducing the risk of business anomalies caused by data changes, and effectively improving the quality of dataset field lifecycle management, ensuring the stable and reliable operation of the overall data system. This solves the problems of high overall operation and maintenance costs, cumbersome operations, and high error rates in existing technologies.

[0011] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a dataset field management method with recoverable unified hash identifiers according to embodiments of this disclosure; Figure 2 This is a schematic diagram of a dataset field management method with recoverable unified hash identifier according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a dataset field management system with recoverable unified hash identifiers according to an embodiment of this application; Figure 4 This is a schematic diagram of a data transmission device provided according to an embodiment of the present disclosure. Detailed Implementation

[0013] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0014] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0015] The terminology used herein, including but not limited to, is open-ended. The term "based on" is at least partially based on. The term "one embodiment" refers to at least one embodiment; the term "another embodiment" refers to at least one additional embodiment; the term "some embodiments" refers to at least some embodiments. Relevant definitions for other terms will be given in the description below.

[0016] It should be noted that the concepts of "first," "second," etc., mentioned in this disclosure are used only to distinguish different devices, modules, or units, and are not used to define the order of functions performed by these devices, modules, or units or their interdependencies.

[0017] It should be noted that the one or more modifications mentioned in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that they should be understood as one or more unless explicitly indicated in the context.

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] The following describes a method for managing dataset fields with recoverable unified hash identifiers, based on embodiments of the present disclosure, with reference to the accompanying drawings. Addressing the issues of cumbersome and error-prone operations mentioned in the background section, this application provides a method for managing dataset fields with recoverable unified hash identifiers. This method generates unified and unique field hash identifiers by employing differentiated hash calculation rules for basic and derived fields. Metadata association and storage are completed based on pre-built dataset tables and dataset field tables. This standardizes field names and derived SQL expression formats, effectively avoiding data corruption caused by field name conflicts and expression modifications. Furthermore, the dedicated hash field identifiers facilitate automatic field recovery, dataset replication, and downstream dataset switching operations, significantly improving field traceability accuracy and data asset reuse efficiency, enhancing data operation and maintenance fault tolerance, reducing business anomaly risks caused by data changes, and effectively improving the quality of full lifecycle management of dataset fields, ensuring the stable and reliable operation of the overall data system. Thus, it solves the problems of high overall operation and maintenance costs, cumbersome operations, and high error propensity in the prior art.

[0020] Specifically, Figure 1 This disclosure provides a method for managing dataset fields with recoverable unified hash identifiers.

[0021] like Figure 1 As shown, the unified hash identifier's data set field management method with recovery capabilities includes the following steps: In step S101, the user-initiated dataset field management operation request is obtained.

[0022] Accordingly, by extracting user identity identifiers, operation type identifiers, and unique identifiers of the target dataset, permission verification and specific operation type identification are completed, effectively avoiding unauthorized operations and intercepting illegal and invalid requests. At the same time, the messy raw request data is organized into standardized operation request objects that conform to preset specifications, unifying the request processing standards. This provides secure, accurate, and standardized pre-processing data support for core processes such as subsequent dataset metadata generation, field unique hash identifier calculation, field metadata storage, automatic field recovery, dataset copying, and downstream dataset switching, ensuring that the entire dataset field management process operates in a compliant, orderly, accurate, and stable manner.

[0023] In this embodiment of the disclosure, obtaining a user-initiated dataset field management operation request includes: obtaining the dataset field management operation request initiated by the user through a front-end interactive interface, extracting the user identity identifier, operation type identifier, and target dataset unique identifier carried in the request; verifying the user's operation permissions on the target dataset based on the user identity identifier, operation type identifier, and target dataset unique identifier, identifying the specific operation type corresponding to the request as field creation, field modification, field deletion, or field restoration, and obtaining the permission verification pass result and operation type identification result; and parsing the field parameter information carried in the request based on the permission verification pass result and operation type identification result to generate a standardized dataset field management operation request object that conforms to a preset processing specification.

[0024] Among them, the target dataset unique identifier refers to a unique identifier used to unambiguously identify and locate a specific dataset within a specific system or globally.

[0025] Correspondingly, the unique identifier of the target dataset enables accurate differentiation and unique positioning of different datasets in massive business scenarios, effectively avoiding data confusion and operation mismatch problems that occur when managing multiple datasets in parallel. It can accurately bind user operations to the corresponding target dataset, assisting in the accurate verification of dataset operation permissions and preventing problems such as unauthorized operations and cross-dataset misoperations. At the same time, as the core link between datasets and field data, this identifier can accurately connect dataset metadata, field hash unique identifiers, field metadata and upstream and downstream dependency data, providing accurate data positioning and association support for the entire process of dataset creation, field normalized hash generation, field storage, field automatic recovery, dataset copying, and downstream dataset switching, ensuring the accuracy, orderliness and stability of the entire dataset field management process.

[0026] In step S102, the dataset field management operation request is parsed to obtain the operation type, target dataset information, and field information to be processed. The dataset table and dataset field table stored in the metadata database are pre-built. The dataset field table is a list of metadata used to define each field in the dataset.

[0027] Correspondingly, the dataset field table, as the core structured carrier storing all field metadata, uses the field's unique hash identifier as the primary key. It establishes a precise mapping relationship based on the dataset's unique identifier associated with the dataset table, and uniformly standardizes the storage of various core information for basic and derived fields. This includes field aliases, original field information, standardized SQL expressions, field dependencies, attribution information, status markers, and time information. This enables the orderly collection, unique identifier binding, and standardized management of all dataset field data. It effectively distinguishes the attribute differences of different types of fields, records complete field metadata and upstream / downstream dependencies, and provides underlying data support for the generation, association, and persistent storage of field unique hash identifiers. It also retains the complete historical state and associated data of fields, accurately supporting core operations such as automatic field recovery, dataset replication, and downstream dataset switching. This avoids problems such as chaotic field management, identifier conflicts, association failures, and inability to trace and recover data, significantly improving the standardization, traceability, compatibility, and fault recovery capabilities of dataset field management.

[0028] In this embodiment of the disclosure, a dataset table and a dataset field table stored in a metadata database are pre-constructed, including: constructing a dataset table and a dataset field table stored in a metadata database; based on the constructed dataset table and dataset field table, defining the dataset table to include auto-incrementing dataset unique identifier, dataset name, owner user unique identifier, creation time, and update time fields, and defining the dataset field table to include field unique identifier, field alias, base field original name, derived field SQL expression, owner user unique identifier, creation time, update time, and status fields; according to the defined dataset table and dataset field table, using the auto-incrementing dataset unique identifier as the primary key of the dataset table, and using the field unique identifier as the primary key of the dataset field table, establishing a mapping relationship between the two tables through the dataset unique identifier.

[0029] Among them, the derived field SQL expression refers to the calculation logic that dynamically generates new fields that are not directly derived from the original data table through SQL code such as operations, functions, or conditional statements.

[0030] Correspondingly, by standardizing the SQL expressions of derived fields and replacing field references with unique field identifiers before generating hash-unique identifiers, the problems of incorrect derived field identifiers and inconsistent definitions caused by changes in the original field names and differences in SQL expression writing formats are effectively avoided. The calculation logic and field dependencies of derived fields are accurately solidified. Based on the standardized SQL expressions and corresponding unique field identifiers, the relationships between derived fields can be automatically adapted and compatible during the dataset replication process, and the deleted fields can be automatically restored. At the same time, it supports seamless switching of definitions in downstream datasets without the need for manual modification of downstream field references and SQL statements. This greatly improves the stability, recoverability, and reusability of dataset field management, and reduces the operational costs and business error risks during data iteration, migration, and maintenance.

[0031] In step S103, when the operation type is dataset creation, dataset metadata is generated based on the target dataset information, the dataset metadata is saved to the dataset table, and an auto-incrementing unique identifier for the dataset is generated.

[0032] Dataset metadata is structured information used to describe the content, structure, source, quality, and access methods of a dataset.

[0033] Correspondingly, the dataset metadata integration solidifies core attributes across all dimensions, including dataset name, business domain affiliation, data source address, query filtering conditions, access permissions, version information, and creation information. These attributes are stored in a standardized and structured manner in the dataset table and bound to a globally unique auto-incrementing dataset identifier. This fully preserves the dataset's source chain, configuration rules, permission system, and lifecycle information, achieving standardized, refined, and unified management of the dataset. It not only effectively distinguishes and locates each dataset, preventing issues such as data confusion, unclear ownership, and disordered configuration, but also provides complete and reliable underlying data support for subsequent core operations such as field association storage, field recovery, dataset replication, and downstream dataset switching. This ensures the stability, traceability, and reusability of the overall dataset architecture, greatly reducing the cost of dataset iteration management and operation and maintenance adaptation.

[0034] In this embodiment of the disclosure, the generation of dataset metadata based on the target dataset information, the saving of the dataset metadata to the dataset table, and the generation of an auto-incrementing unique dataset identifier include: obtaining complete target dataset information corresponding to the dataset creation request initiated by the user; extracting dataset-related metadata information from the target dataset information, wherein the dataset-related metadata information includes dataset name, business domain attribution description, attribution user unique identifier, data source connection address, database table name, data query filtering conditions, role access permission configuration, initial version number, and creator information; integrating the dataset-related metadata information into complete dataset metadata according to the preset standardized structure of the metadata database, persistently saving the dataset metadata to the pre-built dataset table, and triggering the database primary key auto-incrementing mechanism to automatically generate a globally unique auto-incrementing dataset identifier.

[0035] Among them, the auto-incrementing dataset unique identifier refers to the identifier field in the database or dataset that is generated by automatically incrementing values ​​and is used to uniquely distinguish each record.

[0036] Correspondingly, the auto-incrementing dataset unique identifier serves as the primary key of the dataset table. Automatically generated by the database's auto-incrementing mechanism, it possesses global uniqueness, orderliness, and non-repeatability. Requiring no manual coding or configuration, it accurately distinguishes all independent datasets within the system, effectively preventing data confusion and identifier conflicts caused by duplicate dataset names or similar data sources. Simultaneously, this identifier serves as the core association key, establishing a binding mapping between the dataset table and the dataset field table. It precisely links all metadata and subordinate field data of the corresponding dataset, ensuring the rigor and stability of the dataset-field association. This provides reliable index support for the entire process of dataset creation, field binding, dataset replication, downstream caliber switching, and automatic field recovery, enabling traceability, associativity, and controllability throughout the dataset's lifecycle, significantly improving the standardization and accuracy of dataset metadata management.

[0037] In step S104, when the operation type is field creation, if the field to be created is a basic field, the original name of the basic field is normalized and the hash value is calculated to obtain the field's unique identifier. When the field to be created is a derived field, the original SQL expression of the derived field is normalized, and all field names in the normalized SQL expression are replaced with the corresponding stored field unique identifiers. The hash value of the replaced expression is then calculated to obtain the field's unique identifier.

[0038] Among them, the basic field refers to the core attribute unit in a data table or data structure used to store the smallest indivisible business information.

[0039] Correspondingly, the basic fields, as the underlying raw data carriers of the dataset system, standardize the original names and generate unique field identifiers using hash algorithms. This unifies the identifier generation standard for the underlying raw fields, effectively avoiding problems such as field identification errors and matching failures caused by non-standard naming, messy formats, and homonyms in the original fields. The metadata such as the original field names, field types, data sources, and business descriptions stored in the basic fields provide a stable and reliable underlying dependency benchmark for the construction of derived fields and the mapping of SQL expression fields. At the same time, relying on the fixed and unique hash field identifiers, it can stably support core operations such as automatic field recovery, dataset replication, and seamless switching of downstream datasets, ensuring the consistency and traceability of the underlying field structure of the dataset. This reduces the risk of dataset inconsistencies and dependency failures caused by changes in the underlying fields, thus laying a solid foundation for the entire dataset field management mechanism.

[0040] In step S105, the field unique identifier is associated with the corresponding field metadata and stored in the dataset field table. Based on the string field unique identifier in the dataset field table, the field automatic recovery, dataset copying and downstream dataset switching operations are performed.

[0041] Correspondingly, storing the field's unique identifier and metadata in the dataset's field table allows for the complete binding and archiving of field identity information, attribute parameters, dependencies, and status data, forming a standardized field archive. Using the string-based unique identifier as a unified retrieval and association basis, it efficiently supports multiple core operational and maintenance operations. This enables rapid automatic recovery of accidentally deleted fields by retrieving archived metadata, preserving the original data logic and business attributes. Furthermore, it allows for precise matching of corresponding fields during dataset replication, automatically ensuring compatibility with various related dependencies. It also enables smooth switching of downstream datasets without requiring any changes to field references, effectively avoiding issues such as incorrect associations and distorted definitions during field adjustments and data migration. This significantly improves the completeness, flexibility, and reliability of field management, reducing the workload and error probability of data operation and maintenance adjustments.

[0042] In this embodiment of the disclosure, the field unique identifier is associated with and stored in the dataset field table, including: obtaining the generated field unique identifier and the complete metadata information of the corresponding field; based on the obtained field unique identifier and the complete metadata information of the corresponding field, the field type is distinguished as a basic field and a derived field, and the field alias, original name, field type, data source column, and business description of the basic field are organized, and the field alias, processed SQL expression, user-displayed SQL expression, list of dependent field unique identifiers, and business description of the derived field are organized; based on the organized basic field and derived field, the field unique identifier is used as the primary key of the dataset field table, associated and bound with the corresponding field metadata, and stored in the pre-built dataset field table.

[0043] Among them, derived fields refer to fields generated from original data or other existing fields through rules such as calculation, transformation, splicing or logical deduction. Their values ​​are not directly input by the user or collected from the outside, but are dynamically generated based on existing information.

[0044] Correspondingly, derived fields are built upon basic fields and can perform secondary processing, aggregation calculations, and standardization of the original underlying data through custom standardized SQL expressions. This effectively compensates for the shortcomings of basic original fields in adapting to complex business statistics, personalized data filtering, and customized analysis, expanding the data processing capabilities and business adaptability of the dataset. Simultaneously, this method standardizes the original SQL expressions of derived fields and uniformly replaces the fields within the expressions with unique hash field identifiers to generate exclusive unique identifiers. This completely avoids issues such as inconsistent derived field identifiers, drifting calculation methods, and dependency failures caused by differences in SQL writing formats and changes in basic field names. It accurately solidifies the calculation logic and field dependencies of derived fields, enabling stable adaptation to operations such as automatic field recovery, dataset copying, and seamless switching of downstream dataset standards. This ensures the consistency, reusability, and traceability of complex processed field logic, significantly improving the stability and operational efficiency of dataset field management in complex business data scenarios.

[0045] In this embodiment of the disclosure, based on the unique identifier of a string field in the dataset field table, the operations of automatic field recovery, dataset copying, and downstream dataset switching are performed, including: obtaining the unique identifier of the field already stored in the dataset field table, as well as the complete historical metadata, soft deletion status marker, and dependency information of the corresponding field; according to the obtained unique identifier of the string field, historical metadata, soft deletion status marker, and dependency information, the processing actions of automatic recovery after field soft deletion, full metadata copying of the dataset, and switching of the downstream business dataset scope are performed respectively; after the processing actions are performed, the downstream dependencies automatically take effect after field recovery, the derived field associations are automatically compatible during dataset copying, and no field references need to be modified during downstream switching.

[0046] The soft deletion status flag refers to a field or attribute used to identify whether a data record has been logically deleted.

[0047] Correspondingly, adding soft deletion status markers can eliminate the operation mode of directly physically deleting field data. Only the status markers can distinguish between the available and deleted status of a field. The metadata, hash unique identifier, and upstream and downstream dependencies of the deleted field are completely retained. This can effectively avoid the problem of permanent data loss caused by accidental operation, and can also quickly identify the field's usage status based on the status markers, leaving complete original evidence for automatic field recovery. At the same time, it will not disrupt the overall architecture of the dataset and the binding logic between fields. When carrying out operations such as dataset copying and downstream dataset caliber switching, the status markers can be used to identify valid fields, ensuring the normal progress of various business operations. It also facilitates the tracking of field change history, improves the fault tolerance of field management and the integrity of the data system, and reduces the risk of business failure caused by accidental deletion.

[0048] The following will illustrate a specific example of a data set field management method with recoverable unified hash identifiers, including: In big data processing and data analysis scenarios within business intelligence systems, a unified hash identifier-based dataset field management method with recoverability enables stable identification, automatic recovery, lossless replication, and rapid downstream switching of dataset fields. The following, combined with a standardized business implementation example, fully presents the entire execution logic of this method, including core operations, data flow, and identifier generation rules. Figure 2 As shown, all data follows hash calculation specifications and metadata storage rules to ensure the uniqueness, stability and consistency of field identifiers, and the entire lifecycle management of fields can be completed without human intervention.

[0049] I. Core Metadata Database Table Construction: Before executing any management operations, the system pre-constructs two core storage tables in the metadata database: the dataset table and the dataset field table. This table structure forms the data foundation for the entire management method. The two core tables in the metadata storage layer provide persistent support for all operations, and all dataset and field information are stored according to a fixed structure. The dataset table contains five core fields: id (long integer), name (string), user_id (long integer), create_time (date), and update_time (date). The id field stores the unique identifier of the auto-incrementing dataset, the name field stores the dataset name, the user_id field stores the dataset ownership identifier, and the create_time and update_time fields record the creation and update information of the dataset, respectively. The dataset field table contains eight core fields: `id` (string), `alias` (string), `name` (string), `sql_expression` (string), `sql_expression_show` (string), `user_id` (long integer), `create_time` (date), and `update_time` (date). The `id` field stores the unique identifier of the field (a string), the `alias` field stores the field's alias, the `name` field stores the original name of the base field, the `sql_expression` field stores the normalized expression of the derived field, and the `sql_expression_show` field stores the original expression entered by the user. The remaining fields record the field's ownership, creation, and update information. This table structure differs from traditional auto-incrementing ID storage; it uses strings to store the unique identifier of the field, providing a foundation for stable storage of hash identifiers.

[0050] II. Operation Request Acquisition and Parsing: The system acquires user-initiated dataset field management operation requests in real time. It decomposes the request information using standardized parsing rules to obtain the operation type, target dataset information, and fields to be processed. This step is the entry point for the entire management method. The request parsing module transforms user operations into executable system instructions, ensuring all operations are executed based on standardized input. In this embodiment, the user sequentially initiates seven types of operations: dataset creation, basic field creation, derived field creation, field deletion, field recovery, dataset copying, and downstream dataset switching. The target dataset information includes four core categories: dataset name, data source configuration, business domain, and permission configuration. The fields to be processed include five categories: field type, original name, SQL expression, field data type, and data source. After parsing, the system clarifies the execution target of each type of operation. The dataset creation operation requires generating a unique dataset identifier and storing metadata. The field creation operation requires distinguishing between basic and derived fields to generate corresponding hash identifiers. Field deletion and recovery operations rely on hash identifiers to achieve soft deletion and automatic recovery. The dataset copying operation requires retaining the field identifier and updating the dataset ownership. The downstream switching operation only needs to change the dataset identifier to achieve compatibility.

[0051] III. Dataset Creation Operation Execution: When the operation type is determined to be dataset creation, the system generates complete dataset metadata based on the target dataset information, saves the metadata to the dataset table, and generates a unique dataset identifier using an auto-incrementing rule. The dataset creation process generates a stable unique dataset identifier through an auto-incrementing algorithm, laying the foundation for field attribution and subsequent copying operations. In this embodiment, the target dataset is an e-commerce transaction analysis dataset. The system extracts metadata such as the dataset name, data source address, database authentication information, and business domain attribution, writes it to the dataset table, and generates a unique dataset identifier of 1001. This identifier is a long integer auto-incrementing value, globally unique and fixed, serving as the core identity identifier of the dataset. After the dataset metadata is stored, the system completes dataset initialization and can proceed to the field creation stage. All subsequently created fields are assigned to the unique dataset identifier 1001, achieving the association and binding between the dataset and the fields.

[0052] IV. Field Creation Operation Execution: The field creation operation needs to distinguish between basic fields and derived fields, and use different hash identifier generation rules for each. All generated field unique identifiers are of string type and stored in the id field of the dataset field table. The field creation module is the core link in hash identifier generation, and the identifier generation logic for basic fields and derived fields is clear and stable. (I) Basic Field Creation: In this embodiment, two basic fields are created. The original name of the first basic field is user ID. The system first performs normalization processing on the original name, removing leading and trailing spaces, converting it to lowercase letters, and replacing special characters with single underscores, finally obtaining the normalized name user_id. Then, MD5 cryptographic hash calculation is performed on user_id to generate a 32-bit hexadecimal hash value, and the first 10 digits are truncated as the unique identifier of the basic field. The calculation result is b603f8cc92. The original name of the second basic field is order amount. After normalization processing, order_amount is obtained. MD5 hash calculation is performed and the first 10 digits are truncated to generate the field unique identifier ea1c78dd82. The system associates and stores the unique identifiers, original names, field value types, data source tables, and dataset ID1001 of the two basic fields into the dataset field table, thus completing the creation of the basic fields. The same normalized field name always generates the same hash identifier, ensuring the permanent stability of the basic field identifier. (II) Derivative Field Creation In this embodiment, a derivative field for the total transaction amount is created. The user-inputted original SQL expression is sum(order_amount). The system first performs normalization processing on the original SQL expression, converting SQL keywords to uppercase, removing redundant spaces and comments, and standardizing the space format of operators and parentheses, resulting in the normalized expression SUM(order_amount). Then, the field name order_amount in the normalized expression is replaced with the corresponding unique identifier of the basic field ea1c78dd82, generating the replaced expression SUM([ea1c78dd82]). An MD5 hash calculation is performed on this replaced expression, and the first 10 characters are truncated to obtain the unique identifier c3adcd1792 of the derivative field. The system stores the unique identifier, field alias, original SQL expression, normalized expression, list of dependent field IDs, and belonging dataset ID1001 of the derived field in a complete dataset field table. SQL expressions with the same computational logic always generate the same hash identifier, ensuring that the derived field identifier is deeply bound to the computational logic and is unaffected by creation time or operation process.

[0053] V. Field Deletion and Automatic Recovery Operations: Field deletion employs a soft deletion mechanism, while field recovery relies on the stability of hash identifiers to achieve zero-cost automatic repair. The field deletion and recovery processes do not require modification of the identifier; only the field status needs to be changed to complete dependency repair. For soft field deletion, a user initiates a basic field deletion request. The target dataset ID is 1001, and the field to be deleted is the user ID basic field, with a unique identifier of b603f8cc92. The system first performs a downstream dependency check, finding that this field is dependent on a transaction total amount derived field. Since no external business system directly references it, a soft deletion operation is performed. The field's status is marked as deleted in the dataset field table, retaining all metadata and hash identifier information, only hidden from the front-end display. At this point, the derived field becomes invalid due to the deletion of the dependent field. Traditional solutions require manual reconstruction of the dependency, but this method retains the identifier and metadata, reserving a foundation for recovery. For automatic field recovery, a user initiates a field recovery request, re-entering the original name of the basic field, user ID. The system again performs normalization processing on this name to obtain user_id, repeats the MD5 hash calculation process, and still generates the field's unique identifier b603f8cc92. The system retrieves the hash identifier from the dataset field table, matches the deleted field record, and directly restores the field status to valid, clearing the deletion mark and operation information without modifying any metadata, dependencies, or hash identifiers. After restoration, the dependencies of the transaction total amount derived field are automatically repaired, and downstream references immediately return to normal. The entire process requires no manual adjustment of dependency configurations, achieving automated recovery after field deletion.

[0054] VI. Dataset Copying Operation: The dataset copying operation does not require rebuilding field identifiers and dependencies; it only requires updating the dataset ownership identifier. The dataset copying process relies on the immutability of hash identifiers to achieve automatic dependency compatibility, eliminating the manual adjustment costs of traditional copying. When a user initiates a dataset copying request, the source dataset's unique identifier is 1001. The system first reads all metadata of the source dataset, including name, data source, permissions, and version information. Simultaneously, it reads the metadata of all fields under the source dataset, including all information of the basic fields b603f8cc92 and ea1c78dd82, and the derived field c3adcd1792. The system generates a new auto-incrementing dataset unique identifier 2002, copies the source dataset metadata, renames it to "E-commerce Transaction Analysis Dataset Copy," and stores it in the dataset table. Subsequently, it copies all field metadata, keeping the field unique identifiers unchanged, only replacing the dataset ID to which the field belongs from 1001 to 2002, and writes this data to the dataset field table in batches. After the copy is completed, the system automatically verifies the dependencies of the derived fields. Both the dependent field ID and the derived field ID retain their original values, with no issues of dependency breakage, conflict, or missing values. The entire copy process only performs data copying, without any operations such as identifier mapping, expression replacement, or dependency reconstruction. The copying efficiency and completeness are far superior to traditional solutions.

[0055] VII. Downstream Dataset Switching Operation: Downstream business systems include three types of applications: data reports, visualization dashboards, and data interfaces. All of them reference the source dataset ID 1001 and the unique field identifiers b603f8cc92, ea1c78dd82, and c3adcd1792. Switching downstream datasets only requires changing the dataset ID; field references remain unchanged, achieving one-click compatibility and rapid switching. When a business needs to switch to a new dataset, traditional solutions require establishing a field ID mapping relationship and replacing the field references of each downstream business individually, which is prone to omissions and errors. In this method, only the dataset ID referenced by the downstream business system needs to be changed from 1001 to 2002; all field unique identifiers remain unchanged. The system automatically matches the corresponding fields under the new dataset ID 2002. Field references, calculation logic, display configurations, and data calls do not require adjustment. After the switch is completed, downstream businesses can obtain data normally without errors, data anomalies, or dependency failures. If a rollback is needed, simply change the dataset ID back to 1001. The switch and rollback are completed in seconds, significantly reducing downstream adaptation costs and operational risks.

[0056] VIII. Core Data and Technical Effects of Method Execution: In this embodiment, the normalized name of the basic field User ID is user_id, and its unique hash identifier is b603f8cc92; the normalized name of the basic field Order Amount is order_amount, and its unique hash identifier is ea1c78dd82; the normalized replacement expression for the derived field Total Transaction Amount is SUM([ea1c78dd82]), and its unique hash identifier is c3adcd1792; the unique identifier of the source dataset is 1001, and the unique identifier of the new dataset after copying is 2002. All hash identifiers are 10-digit hexadecimal strings truncated after MD5 calculation, and the values ​​are accurate and error-free. This method achieves three core technical effects through a unified hash identifier: first, zero-cost automation of field deletion recovery, where the hash identifier of the reconstructed field is completely consistent with the original identifier, and downstream dependencies are automatically repaired without manual intervention; second, guaranteed integrity of dataset replication, where field identifiers remain unchanged after replication, and derived field dependencies are automatically compatible, with no dependency reconstruction costs; and third, extremely simplified operation for switching downstream datasets, requiring only a change to the dataset ID, without modifying field references, reducing adaptation costs by more than 90%. The entire process relies on the dataset table and dataset field table in the metadata database, combined with the stability of the hash identifier, to completely solve the problems of unstable identifiers, unrecoverable deletions, broken replication dependencies, and complex downstream switching in traditional field management, achieving intelligent, automated, and stable management of dataset fields in big data scenarios.

[0057] In summary, this invention, by differentiating between basic and derived fields through standardized hash generation rules, completely solves many pain points in traditional business intelligence big data field management, such as unstable identifiers, difficulty in recovering deleted fields, dependency breaks during dataset replication, and complex downstream business switching adaptation. It can achieve the uniqueness and long-term stability of dataset field identifiers, and automatically recover fields by relying on soft deletion mechanisms and the immutability of hash identifiers. It ensures the complete compatibility of field information and dependencies during dataset replication, significantly simplifies the downstream business dataset switching process, avoids errors and risks caused by manual operation, and achieves unattended, full-lifecycle field management. This effectively reduces data operation and maintenance and business adaptation costs, and comprehensively improves the stability, integrity, and intelligence level of dataset field management in big data scenarios.

[0058] Figure 3 This is a block diagram of a unified hash identifier dataset field management system with recovery capabilities according to an embodiment of this disclosure.

[0059] like Figure 3 As shown, the unified hash identifier dataset field management system 10 with recovery capability includes: a request acquisition module 100, a request parsing module 200, a dataset creation module 300, a field identifier generation module 400, and a storage operation and maintenance execution module 500.

[0060] The system comprises the following modules: Request Acquisition Module 100, which acquires user-initiated dataset field management operation requests; Request Parsing Module 200, which parses the dataset field management operation requests to obtain the operation type, target dataset information, and field information to be processed, and pre-builds a dataset table and a dataset field table stored in the metadata database; Dataset Creation Module 300, which, when the operation type is dataset creation, generates dataset metadata based on the target dataset information, saves the dataset metadata to the dataset table, and generates an auto-incrementing unique dataset identifier; Field Identifier Generation Module 400, which, when the operation type is field creation, if the field to be created is a basic field, calculates the hash value of the original name of the basic field after normalization to obtain the field's unique identifier; if the field to be created is a derived field, it normalizes the original SQL expression of the derived field, replaces all field names in the normalized SQL expression with the corresponding stored unique field identifier, and calculates the hash value of the replaced expression to obtain the field's unique identifier; and Storage Operation and Maintenance Execution Module 500, which associates the field's unique identifier with the corresponding field metadata and stores it in the dataset field table, and performs automatic field recovery, dataset copying, and downstream dataset switching operations based on the string field unique identifier in the dataset field table.

[0061] It should be noted that the explanation of the aforementioned embodiment of the data set field management method with recoverable unified hash identifier also applies to the data set field management system with recoverable unified hash identifier in this embodiment, and will not be repeated here.

[0062] The unified hash identifier-based dataset field management system with recovery capabilities proposed in this disclosure generates a unified and unique field hash identifier by applying differentiated hash calculation rules to basic fields and derived fields. It relies on pre-built dataset tables and dataset field tables to complete metadata association and storage. This standardizes field names and derived SQL expression formats, effectively avoiding data corruption caused by field name conflicts and expression modifications. Furthermore, it enables automatic field recovery, dataset replication, and downstream dataset switching operations through dedicated hash field identifiers. This significantly improves field traceability accuracy and data asset reuse efficiency, strengthens data operation and maintenance fault tolerance, reduces the risk of business anomalies caused by data changes, and effectively improves the quality of full lifecycle management of dataset fields, ensuring the stable and reliable operation of the overall data system. Therefore, it solves the problems of high overall operation and maintenance costs, cumbersome operations, and high error rates in existing technologies.

[0063] like Figure 4 As shown, the data transmission device may include a processor 401 and a memory 402 storing computer program instructions.

[0064] In some specific embodiments, the processor 401 described above may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0065] Memory 402 may include a mass storage device for information or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway device. In a particular embodiment, memory 402 is a non-volatile solid-state memory. In a particular embodiment, memory 402 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0066] The processor 401 reads and executes computer program instructions stored in the memory 402 to perform the steps of the unified hash identifier-based dataset field management method provided in this embodiment of the present disclosure.

[0067] In one example, the data transmission device may further include a transceiver 403 and a bus 404. Wherein, as... Figure 4 As shown, the processor 401, memory 402 and transceiver 403 are connected via bus 404 and communicate with each other.

[0068] Bus 404 may include hardware, software, or both. For example, and not limitingly, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 404 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0069] This disclosure also provides a computer-readable storage medium that can store a computer program. When the computer program is executed by a processor, the processor implements the dataset field management method with recoverable unified hash identifier provided in this disclosure.

[0070] The aforementioned storage medium may include, for example, a memory 402 containing computer program instructions, which can be executed by the processor 401 of the battery internal resistance detection device to complete the battery internal resistance detection method provided in this embodiment. In some embodiments, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc ROM (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0071] This disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the unified hash identifier dataset field management method with recoverability provided in this disclosure.

[0072] Furthermore, the foregoing explanations and descriptions of the method and apparatus embodiments also apply to the devices, computer-readable storage media, computer program products, and computer programs described above, and will not be repeated here.

[0073] Furthermore, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Moreover, the term "include" is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0074] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0075] All embodiments disclosed herein can be executed individually or in combination with other embodiments, and are all considered to be within the scope of protection claimed by this disclosure.

Claims

1. A method for managing dataset fields with unified hash identifiers and recovery capabilities, characterized in that, include: Retrieve user-initiated requests for dataset field management operations; The dataset field management operation request is parsed to obtain the operation type, target dataset information, and field information to be processed. A dataset table and a dataset field table stored in the metadata database are pre-built. When the operation type is dataset creation, dataset metadata is generated based on the target dataset information, the dataset metadata is saved to the dataset table, and an auto-incrementing unique identifier for the dataset is generated; When the operation type is field creation, if the field to be created is a basic field, the hash value of the original name of the basic field is calculated after normalization to obtain the unique identifier of the field. When the field to be created is a derived field, the original SQL expression of the derived field is normalized, and all field names in the normalized SQL expression are replaced with the corresponding stored unique identifier of the field. The hash value of the replaced expression is calculated to obtain the unique identifier of the field. The unique identifier of the field is associated with the corresponding field metadata and stored in the dataset field table. Based on the unique identifier of the string field in the dataset field table, the automatic field recovery, dataset copying and downstream dataset switching operations are performed.

2. The method for managing dataset fields with recoverable unified hash identifiers according to claim 1, characterized in that, Retrieve user-initiated dataset field management operation requests, including: Get the dataset field management operation request initiated by the user through the front-end interactive interface, and extract the user identity identifier, operation type identifier and target dataset unique identifier carried in the request; Based on the user identity identifier, operation type identifier, and unique identifier of the target dataset, the user's operation permissions on the target dataset are verified, and the specific operation type corresponding to the request is identified as field creation, field modification, field deletion, or field restoration, thus obtaining the permission verification pass result and the operation type identification result. By combining the permission verification result with the operation type identification result, the field parameter information carried in the request is parsed to generate a standardized dataset field management operation request object that conforms to the preset processing specifications.

3. The method for managing dataset fields with recoverable unified hash identifiers according to claim 1, characterized in that, The dataset table and dataset field table, which are pre-built and stored in the metadata database, include: Construct dataset tables and dataset field tables to be stored in the metadata database; Based on the constructed dataset table and dataset field table, the dataset table is defined to include auto-incrementing dataset unique identifier, dataset name, owner user unique identifier, creation time and update time fields, and the dataset field table is defined to include field unique identifier, field alias, base field original name, derived field SQL expression, owner user unique identifier, creation time, update time and status fields. Based on the defined dataset table and dataset field table, the auto-incrementing dataset unique identifier is used as the primary key of the dataset table, and the field unique identifier is used as the primary key of the dataset field table, thus establishing a mapping relationship between the two tables through the dataset unique identifier.

4. The method for managing dataset fields with recoverable unified hash identifiers according to claim 1, characterized in that, Generate dataset metadata based on the target dataset information, save the dataset metadata to the dataset table, and generate an auto-incrementing unique dataset identifier, including: Obtain complete target dataset information corresponding to the dataset creation request initiated by the user; Extract dataset-related metadata from the target dataset information, including dataset name, business domain attribution description, unique identifier of the attributing user, data source connection address, database table name, data query filtering conditions, role access permission configuration, initial version number, and creator information. Based on the metadata information related to the dataset, it is integrated into complete dataset metadata according to the standardized structure preset by the metadata database. The dataset metadata is then persistently saved to the pre-built dataset table, triggering the database primary key auto-increment mechanism to automatically generate a globally unique auto-incrementing dataset identifier.

5. The method for managing dataset fields with recoverable unified hash identifiers according to claim 1, characterized in that, The unique identifier of the field is associated with the corresponding field metadata and stored in the dataset field table, including: Obtain the unique identifier of the generated field and the complete metadata information of the corresponding field; Based on the obtained field unique identifiers and corresponding complete metadata information, the field types are distinguished as basic fields and derived fields. The field aliases, original names, field types, data source columns, and business descriptions of the basic fields are organized, and the field aliases, processed SQL expressions, user-displayed SQL expressions, lists of dependent field unique identifiers, and business descriptions of the derived fields are organized. Based on the completed basic and derived fields, the unique identifier of each field is used as the primary key of the dataset field table, associated and bound with the corresponding field metadata, and stored in the pre-built dataset field table.

6. The method for managing dataset fields with recoverable unified hash identifiers according to claim 1, characterized in that, Based on the unique identifier of the string field in the dataset field table, perform automatic field recovery, dataset copying, and downstream dataset switching operations, including: Retrieve the unique identifiers of the fields already stored in the dataset field table, as well as the complete historical metadata, soft deletion status markers, and dependency information of the corresponding fields; Based on the obtained string field unique identifier, historical metadata, soft deletion status flag and dependency information, perform the following processing actions respectively: automatic recovery after field soft deletion, full metadata copying of the dataset, and switching of the downstream business dataset caliber. After the processing action is performed, downstream dependencies will automatically take effect after field restoration, the relationships between derived fields will be automatically compatible when the dataset is copied, and no field references need to be modified when switching downstream.

7. A dataset field management system with unified hash identifiers and recovery capabilities, characterized in that, include: The request retrieval module is used to retrieve user-initiated requests for dataset field management operations. The request parsing module is used to parse the dataset field management operation request to obtain the operation type, target dataset information and field information to be processed, and to pre-build the dataset table and dataset field table stored in the metadata database; The dataset creation module is used to generate dataset metadata based on the target dataset information when the operation type is dataset creation, save the dataset metadata to the dataset table, and generate an auto-incrementing unique identifier for the dataset. The field identifier generation module is used to, when the operation type is field creation, if the field to be created is a basic field, calculate the hash value after normalizing the original name of the basic field to obtain the unique identifier of the field; when the field to be created is a derived field, normalize the original SQL expression of the derived field, replace all field names in the normalized SQL expression with the corresponding stored unique identifier of the field, and calculate the hash value of the replaced expression to obtain the unique identifier of the field. The storage operation and maintenance execution module associates the unique identifier of the field with the corresponding field metadata and stores it in the dataset field table. Based on the unique identifier of the string field in the dataset field table, it performs automatic field recovery, dataset copying and downstream dataset switching operations.

8. A data transmission device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the dataset field management method with recoverable unified hash identifier as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement a dataset field management method with recoverable capabilities using the unified hash identifier as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements a dataset field management method with recoverable unified hash identifier as described in any one of claims 1 to 6.

Images